custom tasks

[isspek]

Dear Team,

I would like to use this library to evaluate LLMs on our custom tasks. I couldn't find the documentation about adding new tasks. Where can I find it?

Kind regards,
Ipek

[kargibora]

Dear Ipek,

Thank you for reaching out. You did not miss the documentation: the current version documents the built-in tasks, but we do not yet have a guide or public registration API for custom tasks.

If your custom task is a single-turn instruction dataset that should use JudgeArena's existing pairwise generate-and-judge workflow, it can currently be added with a small code extension.

Required dataset fields

A custom instruction dataset must contain the following fields:

instruction_index,instruction
0,Write a short summary of this text.
1,Explain why the sky appears blue.

• instruction_index: A unique, non-null identifier for each example.
• instruction: The non-null prompt given to the candidate and baseline models.

The loader must return a pandas.DataFrame containing these columns:

import pandas as pd


def load_my_task() -> pd.DataFrame:
    return pd.DataFrame(
        {
            "instruction_index": [0, 1],
            "instruction": [
                "Write a short summary of this text.",
                "Explain why the sky appears blue.",
            ],
        }
    )

Add the custom task to load_instructions() in judgearena/instruction_dataset/__init__.py. For example, if the dataset is stored at $JUDGEARENA_DATA/tables/instructions/my-task.csv:

elif dataset == "my-task":
    from judgearena.paths import data_root, read_df

    df_instructions = read_df(
        data_root / "tables" / "instructions" / "my-task.csv"
    )

You can then run it through the existing pairwise pipeline:

judgearena \
  --task my-task \
  --model.name VLLM/organization/candidate-model \
  --model.baseline VLLM/organization/baseline-model \
  --judge.model OpenRouter/provider/judge-model

For a custom task, --model.baseline is currently required because no default baseline is registered.

Optional existing model outputs

If the dataset already provides model completions, store them at $JUDGEARENA_DATA/tables/model_outputs/my-task.csv.zip. JudgeArena uses OPENJURY_DATA as a legacy fallback and ~/judgearena-data/ when neither environment variable is set.

The compressed CSV must contain:

instruction_index,model,output
0,VLLM/organization/baseline-model,Existing response for instruction 0
1,VLLM/organization/baseline-model,Existing response for instruction 1

• instruction_index: Must match the corresponding instruction.
• model: Must exactly match the corresponding CLI model value, including its backend prefix, such as VLLM/organization/baseline-model.
• output: The existing model response. output is the correct on-disk column name; JudgeArena converts it internally to completion after loading.

Existing outputs are optional. If they are not provided, JudgeArena generates responses using --model.name and --model.baseline.

On the current main branch, try_load_dataset_completions() still attempts to contact the built-in JudgeArena Hugging Face dataset before checking this local output path. A fully local custom-task integration should also adjust that function to skip download_hf() for the custom task and check the local file directly.

If your task requires a different conversation format, judge prompt, parser, or scoring method, it currently requires a benchmark-specific integration rather than only a dataset loader.

We agree that this extension path should be documented and simplified. As also documented in #74, we are trying to improve this extension path.

If you can share the expected format of your custom task, we can suggest the most suitable integration path.

Kind regards,

The JudgeArena Team