Proposal: ghostrun-openeval-adapter (export RunLog/TestRecord as an EvalPort ResultSet)

[adhabnr-ux]

Hi! I really like the RunLog / TestRecord / AssertionRecord design in ghostrun/runlog.py — recording every ghostrun.expect(...) call against the current test with kind, criterion, passed, reason, and output_index, snapshotted per run with to_json/from_json, is a genuinely portable result format, not just an internal cache format. That's rare enough among small eval tools that it seemed worth flagging.

What EvalPort is: EvalPort is an open JSON interchange format for LLM eval data — a TestCase/Grader/EvalSuite for what to test and how, and a ResultSet for what happened when you ran it. The idea is a RunLog produced by ghostrun, a report from DeepEval, and a trace from Promptfoo can all be diffed/compared/archived in one shape instead of three incompatible ones. Full spec: spec/SPEC.md.

Why ghostrun maps cleanly: your RunLog.tests: Dict[str, TestRecord] is structurally almost exactly an EvalPort ResultSet.results array already —

• RunLog (name, created, label, tests) → ResultSet (run_id, started_at, suite_id, results)
• TestRecord (test_id, outputs, assertions, outcome, .passed) → Result (test_case_id, actual_output, grader_results, passed)
• AssertionRecord (kind, criterion, passed, reason, output_index) → GraderResult (grader_id, type, score, passed, reason, metadata)

A standalone ghostrun-openeval-adapter package (no changes needed to ghostrun core — just reads the public dataclasses) could look roughly like:

# ghostrun_openeval_adapter/__init__.py
import uuid
from ghostrun.runlog import RunLog, TestRecord, AssertionRecord

def to_openeval(log: RunLog) -> dict:
    """ghostrun RunLog -> EvalPort ResultSet."""
    results = []
    for test_id, rec in sorted(log.tests.items()):
        grader_results = [
            {
                "grader_id": a.kind,
                "type": "llm_judge" if a.kind not in
                    ("contains", "does_not_contain", "is_valid_json") else "custom",
                "score": 1.0 if a.passed else 0.0,
                "passed": a.passed,
                "reason": a.reason,
                "metadata": {"criterion": a.criterion, "output_index": a.output_index},
            }
            for a in rec.assertions
        ]
        results.append({
            "test_case_id": test_id,
            "actual_output": rec.outputs[-1] if rec.outputs else "",
            "grader_results": grader_results,
            "passed": rec.passed,
            "metadata": {"outcome": rec.outcome, "outputs": rec.outputs},
        })
    return {
        "version": "1.0.0",
        "suite_id": log.name,
        "run_id": f"{log.name}_{uuid.uuid4().hex[:8]}",
        "started_at": log.created,
        "runner": {"name": "ghostrun", "version": "0.1.5"},
        "results": results,
    }

def from_openeval(result_set: dict) -> RunLog:
    """Rehydrate a foreign ResultSet as a RunLog, e.g. to run `ghostrun diff`
    against a run produced by a different eval framework."""
    log = RunLog(name=result_set["run_id"], label=result_set.get("suite_id", ""))
    for r in result_set["results"]:
        rec = log.ensure_test(r["test_case_id"])
        if r.get("actual_output"):
            rec.outputs.append(r["actual_output"])
        for gr in r.get("grader_results", []):
            rec.assertions.append(AssertionRecord(
                kind=gr["type"],
                criterion=gr.get("metadata", {}).get("criterion", gr["grader_id"]),
                passed=gr["passed"],
                reason=gr.get("reason", ""),
            ))
        rec.outcome = "passed" if r["passed"] else "failed"
    return log

A couple of open questions I don't want to guess the answer to on your behalf:

• AssertionRecord.kind (e.g. contains_intent, tone_is) doesn't map onto EvalPort's well-known grader type enum (exact_match, llm_judge, etc.) 1:1. The spec's custom/unrecognized-type path requires a params.handler string for round-tripping, so the adapter would probably want to emit type: "ghostrun_<kind>" with a handler rather than force-fitting into llm_judge. Curious whether that reads right to you.
• regression.py's Comparison/AssertionDelta (regression/fix/stable classification between two RunLogs) doesn't have a direct EvalPort counterpart today — that'd stay ghostrun-native rather than round-tripping, at least in a v1.

Would a PR adding this as a separate ghostrun-openeval-adapter package (own pyproject.toml, depends on ghostrun as a normal dependency, doesn't touch this repo's own package) be something you'd want linked from the README, or would you rather it just live independently and reference ghostrun? No pressure either way — mostly wanted to check the mapping made sense to you before writing it, since I only skimmed the code and don't want to guess at your intent for kind wrong.

(For context: I maintain EvalPort and I'm doing this same kind of outreach to a handful of eval/observability projects whose result data model looks portable — not just yours. Happy to close this out if it's not a direction you're interested in.)