Improve contamination judge

[rank-and-file]

Currently the judge has some problems, e.g. it often flags GSM8K runs when they shouldn't be flagged.

Therefore we need to improve the judge in the following ways:

1. create a script to decide if a dataset is contaminated (see Olmo 3 paper section 3.5.3, and https://github.com/allenai/decon package). The script should accept any document and automatically check if it contains evaluation data for the current benchmark. This is for the judge to use.
2. also use the trace for the judge. Right now the judge only uses the code generated by the agent. Also here, a tool or agent skill would be useful for the judge to search through the trace.

[rank-and-file]

@hrdkbhatnagar
would you like to work on (1) and I work on (2) ?

[rank-and-file]

In general we should get this ready before harbor integration (because we want people to be able to trust the judge and not need to double check)

[hrdkbhatnagar]

Good idea -- agreed this could be fixed before Harbor.

I'm currently heads down on the Harbor adapter (pre-agent hook) work, so feel free to take (1) and I can review once you have something up. Happy to sync on it once I free up

[rank-and-file]

create a script to decide if a dataset is contaminated

I'm working on this

[rank-and-file]

Also output justification of judge in json

[hrdkbhatnagar]

fixed by #50