Skip to content

CI: wire the Librarian eval harness in as a merge gate (explicit 100% + optional recall@k) #956

Description

@PRAteek-singHWY

Follow-up from @northdpole's reviews on #925 and #937 (Module C - The Librarian).

The gap: the eval harness (scripts/evaluate_librarian.py) already fails loudly on an explicit-slice regression - it exits non-zero if the "explicit" slice drops below 100% on the golden dataset. The problem is nothing in CI actually runs it. make test only runs the unit tests, so today that 100% gate exists on paper but isn't enforced. A change could quietly break the explicit slice and still merge green.

What we'd like to do: wire the harness into CI so it guards merges automatically.

  • Add a small subprocess test (or a Makefile target) that runs evaluate_librarian.py against the committed golden dataset (application/tests/librarian/fixtures/golden_dataset.json). If the explicit slice isn't at 100%, the check fails and blocks the merge. This runs fully offline - no API key needed.
  • Later, extend the same setup to cover the C.1 retriever's recall@k. That part needs live embeddings (a populated cache DB + an embedding API key), so it can either be an optional CI job or a documented manual pre-merge check.

The offline explicit gate is the priority since it's deterministic and needs no credentials.

Scope: Module C. Not blocking W1–W3 - this just makes sure the harness is doing its job before the heavier ML slices (W4+) land.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions