From c38d269fec2b70169703855dc2f14dee2abc5014 Mon Sep 17 00:00:00 2001 From: Jason Date: Tue, 20 Oct 2026 08:30:23 -0500 Subject: [PATCH 1/4] docs(evals): pre-register the selector judge cut-precision measurement (#1579) --- .../CUT_PRECISION_PREREGISTRATION.md | 122 ++++++++++++++++++ 1 file changed, 122 insertions(+) create mode 100644 evals/concept_model_screen/CUT_PRECISION_PREREGISTRATION.md diff --git a/evals/concept_model_screen/CUT_PRECISION_PREREGISTRATION.md b/evals/concept_model_screen/CUT_PRECISION_PREREGISTRATION.md new file mode 100644 index 00000000..4e5214cc --- /dev/null +++ b/evals/concept_model_screen/CUT_PRECISION_PREREGISTRATION.md @@ -0,0 +1,122 @@ +# Pre-registration: does the selector judge's cut buy precision? (#1579, step 1) + +**Status: REGISTERED 2026-10-20, before any `gpt-oss:20b` call was sent for +this study and before any cut or kept object was labelled.** A change after +the commit that adds this file is a new registration, not an edit to this +one. + +## Question + +#1571 showed that every selector judge (the extractor judging itself, or the +packaged `gemma4:26b-a4b`) costs window lift. Window lift measures coverage; +the judge exists to buy precision. Are the objects the judge **cuts** about +as source-supported as the objects it **keeps**? If so, the cut costs lift +without buying precision. + +This is step 1 of #1579 only. It measures; it changes no production code and +no default. Any loosening of the selection needs its own ADR afterwards. + +## No generation by candidate models + +- Only the #1571 replay records in + `~/Dev/Projects/openkos-eval-runs/judge-study/records/` and the #1557 + corpus under `~/Dev/Projects/openkos-eval-runs/concept-screen/` are read. + Neither directory is written. +- The only model called is `gpt-oss:20b`, the #1557 guardrail judge, with the + #1557 G1 prompt and parser (`screen_judge.py`, unchanged). No extraction, + no selector-judge call, no embedding. +- New judge calls and their cache go to + `~/Dev/Projects/openkos-eval-runs/cut-precision/`. + +## Population + +- **Primary:** round 1, run 1, pool DA (ES2015a, IS1004b, TS3009c, ES2015d, + IS1004d), all five extractors (`qwen3:8b`, `qwen3.6:35b-a3b`, + `gemma4:26b-a4b`, `qwen3:14b`, `qwen3.6:27b`): 25 meeting-runs. +- **Secondary (reported only):** run 2 for `qwen3:8b` and `qwen3.6:27b`, the + only extractors with two runs: 10 more meeting-runs. +- **Objects.** For each record, the `ceiling` stage (the selector judge's + input) is partitioned, per judge arm, into KEPT (present in that arm's + `judge` stage) and CUT (the rest). An object is matched by + (type, title, description, window), with multiplicity. Only objects that + carry a window are scored, as in #1557's G1 (the source excerpt is that + window). Objects with no window are counted and excluded. +- **Arms.** `self` (the extractor as judge) and `fixed` (`gemma4:26b-a4b`), + each at seed index 0 for the decision. `self` seed 0 is the recorded #1557 + judge stage, `fixed` seed 0 is the #1571 replay. Seed indices 1 and 2 are + reported as a stability check (point estimates only). For + `gemma4:26b-a4b` as an extractor the two arms are the same sample and are + reported once. + +## Measure + +- **One label per ceiling object.** Per record and per window, all window + ceiling objects go to one G1 call (`gpt-oss:20b`, prompt and parser from + `screen_judge.py`: `supported | partial | unsupported`; seed 0, one retry + at seed 1000 on an unusable reply; a window unusable twice is excluded and + counted). Because the same ceiling is the input of every arm, one call + labels the objects for both arms and every seed, so cut and kept are + judged in the same call, with the same companions. This differs from #1557, + which labelled the final objects of a window; the absolute levels are + therefore not compared with #1557's 0.927 / 0.903, only cut with kept. +- **Precision** = supported / all labelled, pooled (strict, as #1557). + Reported beside it, not decided on: lenient precision + (supported + partial) / all labelled. +- **Statistic.** D = precision(KEPT) - precision(CUT), pooled over windows. + Interval: the two-stage cluster bootstrap the screen uses (meetings, then + windows within a drawn meeting; 10,000 resamples; seed 42; percentile 95%), + on per-window (kept supported, kept labelled, cut supported, cut labelled), + the paired form of #1557's `pooled_precision_delta`. +- Counts printed as `n of TOTAL`: labelled objects of all window objects, + windows labelled of all windows, cut and kept sizes. + +## Decision rule + +Per judge arm (`self`, `fixed`), on the primary population pooled over the +five extractors, seed index 0. Margin 0.05, the same as #1557's G1 tolerance. + +0. **Insufficient** if fewer than 30 CUT objects are labelled in the arm. + No verdict. +1. **The cut buys precision** if the lower bound of D is above +0.05: kept + objects are supported by more than five points over cut ones. +2. **The cut is too strict** if the upper bound of D is below +0.05 **and** + the point precision of the CUT objects is at least 0.80: cut objects are + mostly supported and not clearly worse than kept ones. +3. **Inconclusive** otherwise (the interval straddles +0.05, or cut + objects are both not clearly worse than kept ones and below 0.80). + +Overall: **too strict** if both arms say rule 2; **buys precision** if both +say rule 1; otherwise **mixed or inconclusive**, with the pattern described, +not decided. Per-extractor numbers (and the secondary runs) are reported with +their intervals and are not decisive. + +Consequences, stated before the data: + +- Too strict: #1579 step 2 proposes loosening the selection (drop only + unsupported or duplicate candidates). That is a proposal and needs an ADR. +- Buys precision: the lift cost is recorded as the price of precision, and + #1579 is closed with that record. +- Mixed or inconclusive: recorded as such. No default changes. + +**Limits stated in advance.** One judge model with its own noise; one label +per object (no repeat labelling); a window-only source (an object whose +support lies in another window is read as unsupported); the strict and +lenient views can differ. Windows whose labels are unusable twice are +excluded; if more than 10% of windows are excluded, the run is reported as +degraded and not decided. + +## Forecast + +- Windows to label (windows with at least one windowed ceiling object, from + the `self` seed-0 records): primary 154 (`qwen3:8b` 22, `qwen3.6:35b-a3b` + 28, `gemma4:26b-a4b` 34, `qwen3:14b` 37, `qwen3.6:27b` 33), secondary 62 + (`qwen3:8b` 26, `qwen3.6:27b` 36): **216 calls**. +- #1557 recorded G1 calls at 4.5 to 5.6 s. Ceiling batches are larger than + final batches, so the forecast uses 8 s per call (upper estimate 16 s). +- 216 calls x 8 s = about 29 min, plus one model load of about 1 min: + **about 30 min** (upper estimate about 1 h). `gpt-oss:20b` is the only + model resident. This is well under the 3 h target, so the population is not + reduced. +- If the first 20 calls average more than 25 s, the secondary population is + dropped and the primary alone is run. +- Scoring is model-free. From 11a852a1aaf10403304f6cc41e61a587dedb16d1 Mon Sep 17 00:00:00 2001 From: Jason Date: Tue, 20 Oct 2026 08:32:35 -0500 Subject: [PATCH 2/4] feat(evals): add the selector judge cut-precision harness (#1579) --- evals/concept_model_screen/cut_precision.py | 531 ++++++++++++++++++++ 1 file changed, 531 insertions(+) create mode 100644 evals/concept_model_screen/cut_precision.py diff --git a/evals/concept_model_screen/cut_precision.py b/evals/concept_model_screen/cut_precision.py new file mode 100644 index 00000000..03e0ac59 --- /dev/null +++ b/evals/concept_model_screen/cut_precision.py @@ -0,0 +1,531 @@ +"""G1 precision of the objects the selector judge cuts vs keeps (#1579, step 1). + +Reads the #1571 replay records (`judge_study.py`) and the #1557 corpus; never +writes either. Only `gpt-oss:20b`, the #1557 guardrail judge, is called, with +the #1557 G1 prompt and parser. No extraction and no selector-judge call is +made. Read `CUT_PRECISION_PREREGISTRATION.md` first. + +Subcommands: + +- `label` one G1 call per record window over every windowed `ceiling` object + (the selector judge's input), cached and resumable. +- `score` precision of KEPT and CUT objects per judge arm, the paired + bootstrap interval of their difference, and the registered rule. + +Model-free check: `--self-test`. +""" + +from __future__ import annotations + +import argparse +import collections +import json +import pathlib +import sys +import time +from collections.abc import Callable, Mapping, Sequence +from typing import Any, Final + +HERE = pathlib.Path(__file__).resolve().parent +REPO_ROOT = HERE.parents[1] +sys.path.insert(0, str(REPO_ROOT / "src")) +sys.path.insert(0, str(HERE)) +sys.path.append(str(REPO_ROOT / "evals")) + +import judge_study as js # noqa: E402 +import screen_judge as sj # noqa: E402 +import screen_metrics as sm # noqa: E402 +import screen_pipeline as sp # noqa: E402 + +Obj = dict[str, Any] + +OUT_DIR: Final = pathlib.Path.home() / "Dev/Projects/openkos-eval-runs/cut-precision" +STUDY_DIR: Final = js.OUT_DIR +SCREEN_DIR: Final = js.SCREEN_DIR +ARMS: Final = ("self", "fixed") +SEEDS: Final = js.SEED_INDICES +MARGIN: Final = 0.05 +"""The registered margin, the same as #1557's G1 tolerance.""" +MIN_CUT: Final = 30 +"""Fewer labelled cut objects than this in an arm: no verdict.""" +CUT_FLOOR: Final = 0.80 +"""Rule 2 also needs the cut objects' point precision at least this.""" +MAX_EXCLUDED_SHARE: Final = 0.10 +"""More windows than this share unlabelled: the run is degraded.""" + + +# --- pure ---------------------------------------------------------------------- + + +def _key(obj: Mapping[str, Any]) -> tuple[Any, ...]: + return (obj["type"], obj["title"], obj["description"], obj.get("window")) + + +def partition( + ceiling: Sequence[Mapping[str, Any]], kept: Sequence[Mapping[str, Any]] +) -> list[bool]: + """For each ceiling object, in order: was it kept. Matched by (type, + title, description, window) with multiplicity, so a repeated object is + kept as many times as the judge stage holds it.""" + remaining = collections.Counter(_key(o) for o in kept) + flags: list[bool] = [] + for obj in ceiling: + key = _key(obj) + if remaining[key] > 0: + remaining[key] -= 1 + flags.append(True) + else: + flags.append(False) + return flags + + +def windowed(ceiling: Sequence[Mapping[str, Any]]) -> dict[int, list[int]]: + """window index -> ceiling indices, for objects that carry a window.""" + out: dict[int, list[int]] = {} + for i, obj in enumerate(ceiling): + if obj.get("window") is not None: + out.setdefault(int(obj["window"]), []).append(i) + return out + + +Item = tuple[int, int, int, int] + + +def window_item( + labels: Mapping[int, str], flags: Sequence[bool], *, lenient: bool +) -> Item: + """(kept supported, kept labelled, cut supported, cut labelled) for one + window. `labels` maps a ceiling index to its label.""" + good = {"supported", "partial"} if lenient else {"supported"} + ks = kt = cs = ct = 0 + for index, label in labels.items(): + if flags[index]: + kt += 1 + ks += label in good + else: + ct += 1 + cs += label in good + return ks, kt, cs, ct + + +def classify( + delta: Mapping[str, float], cut_precision: float, cut_labelled: int +) -> str: + """The registered rule for one judge arm. `delta` is precision(kept) - + precision(cut) as {point, low, high}.""" + if cut_labelled < MIN_CUT: + return "insufficient" + if delta["low"] > MARGIN: + return "buys_precision" + if delta["high"] < MARGIN and cut_precision >= CUT_FLOOR: + return "too_strict" + return "inconclusive" + + +def overall(labels: Sequence[str]) -> str: + if labels and all(x == "too_strict" for x in labels): + return "too_strict" + if labels and all(x == "buys_precision" for x in labels): + return "buys_precision" + return "mixed_or_inconclusive" + + +def degraded(excluded: int, total: int) -> bool: + return total > 0 and excluded / total > MAX_EXCLUDED_SHARE + + +# --- label --------------------------------------------------------------------- + + +def label_path( + out_dir: pathlib.Path, model: str, meeting: str, run: int +) -> pathlib.Path: + return out_dir / "labels" / model.replace(":", "_") / meeting / f"r{run}.json" + + +def cmd_label(args: argparse.Namespace) -> int: + from harness_stamp import build_stamp, model_identity + + rcs = js._screen() + corpus = rcs.Corpus(args.screen_dir) + ident = model_identity(sj.JUDGE_MODEL, args.host) + stamp = args.out_dir / "stamp.json" + stamp.parent.mkdir(parents=True, exist_ok=True) + if not stamp.is_file(): + stamp.write_text( + json.dumps( + build_stamp( + model=sj.JUDGE_MODEL, + prompts={"screen/g1": sj.G1_SYSTEM}, + host=args.host, + ), + indent=2, + ) + + "\n", + encoding="utf-8", + ) + print(f"judge {sj.JUDGE_MODEL} digest {ident.get('digest')}", flush=True) + cache = sp.ChatCache(args.out_dir / "cache" / "judge") + plan = [(m, r) for m, runs in js.PRIMARY for r in runs] + if not args.primary_only: + plan += [(m, r) for m, runs in js.SECONDARY for r in runs if r != 1] + warmed = False + calls = 0 + for model, run in plan: + if args.models and model not in args.models: + continue + for meeting in rcs.data.POOL_DA: + path = label_path(args.out_dir, model, meeting, run) + if path.is_file(): + continue + src = js.replay_path(args.study_dir, "self", model, meeting, run, 0) + record = json.loads(src.read_text(encoding="utf-8")) + ceiling = record["stages"]["ceiling"] or [] + _t, wins, _g = corpus.meeting(meeting) + if not warmed: + print( + f"warm-up {sj.JUDGE_MODEL}: {rcs.warm_up(sj.JUDGE_MODEL, args.host):.1f} s", + flush=True, + ) + warmed = True + rows: list[dict[str, Any]] = [] + for window, indices in sorted(windowed(ceiling).items()): + entries = [rcs.unit_text(ceiling[i]) for i in indices] + labels, wall = rcs.judge_call( + cache, + args.host, + sj.g1_messages(wins[window].text, entries), + len(entries), + sj.G1_LABELS, + ) + calls += 1 + rows.append( + { + "window": window, + "indices": indices, + "labels": labels, + "wall_s": wall, + } + ) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps({"windows": rows}, ensure_ascii=False) + "\n", + encoding="utf-8", + ) + failed = sum(1 for r in rows if r["labels"] is None) + wall_total = sum(r["wall_s"] for r in rows) + print( + f"{time.strftime('%H:%M:%S')} {model} {meeting} r{run}: " + f"{len(rows)} windows, {failed} unparsed, {wall_total:.0f} s " + f"({calls} calls so far)", + flush=True, + ) + return 0 + + +# --- score --------------------------------------------------------------------- + + +def _load(path: pathlib.Path) -> dict[str, Any]: + loaded = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(loaded, dict): + raise SystemExit(f"{path}: not an object") + return loaded + + +def _stat(items: Sequence[Item]) -> float: + return sm.pooled_precision_delta(items) + + +def _summarise(clusters: Mapping[str, list[Item]]) -> dict[str, Any]: + flat = [i for items in clusters.values() for i in items] + ks = sum(i[0] for i in flat) + kt = sum(i[1] for i in flat) + cs = sum(i[2] for i in flat) + ct = sum(i[3] for i in flat) + out: dict[str, Any] = { + "kept": {"supported": ks, "labelled": kt}, + "cut": {"supported": cs, "labelled": ct}, + "windows": len(flat), + } + if kt and ct: + iv = sm.two_stage_bootstrap({k: v for k, v in clusters.items() if v}, _stat) + out["delta"] = {"point": iv.point, "low": iv.low, "high": iv.high} + return out + + +def gather( + study_dir: pathlib.Path, + out_dir: pathlib.Path, + models: Sequence[str], + runs: Sequence[int], + arm: str, + seed: int, + *, + lenient: bool, + meetings: Sequence[str], +) -> tuple[dict[str, list[Item]], dict[str, int]]: + """Per-meeting window items for one arm and seed, plus counts.""" + clusters: dict[str, list[Item]] = {} + counts = { + "windows": 0, + "windows_unlabelled": 0, + "objects_windowed": 0, + "objects_no_window": 0, + "records_skipped": 0, + } + for model in models: + for meeting in meetings: + for run in runs: + lab = _load(label_path(out_dir, model, meeting, run)) + base = _load(js.replay_path(study_dir, "self", model, meeting, run, 0)) + ceiling = base["stages"]["ceiling"] or [] + rec = _load(js.replay_path(study_dir, arm, model, meeting, run, seed)) + if not js.same_objects(rec["stages"]["ceiling"], ceiling): + counts["records_skipped"] += 1 + continue + flags = partition(ceiling, rec["stages"]["judge"] or []) + counts["objects_no_window"] += sum( + 1 for o in ceiling if o.get("window") is None + ) + for row in lab["windows"]: + counts["windows"] += 1 + counts["objects_windowed"] += len(row["indices"]) + if row["labels"] is None: + counts["windows_unlabelled"] += 1 + continue + labels = dict(zip(row["indices"], row["labels"], strict=True)) + clusters.setdefault(meeting, []).append( + window_item(labels, flags, lenient=lenient) + ) + return clusters, counts + + +def cmd_score(args: argparse.Namespace) -> int: + rcs = js._screen() + meetings = list(rcs.data.POOL_DA) + out: dict[str, Any] = {"primary": {}, "secondary": {}, "stability": {}} + prim_models = [m for m, _r in js.PRIMARY] + for arm in ARMS: + for lenient in (False, True): + tag = "lenient" if lenient else "strict" + cl, counts = gather( + args.study_dir, args.out_dir, prim_models, (1,), arm, 0, + lenient=lenient, meetings=meetings, + ) # fmt: skip + row = _summarise(cl) + row["counts"] = counts + out["primary"].setdefault(arm, {})[tag] = row + for model in prim_models: + cl_m, c_m = gather( + args.study_dir, args.out_dir, [model], (1,), arm, 0, + lenient=lenient, meetings=meetings, + ) # fmt: skip + r = _summarise(cl_m) + r["counts"] = c_m + out["primary"][arm].setdefault("by_extractor", {}).setdefault( + model, {} + )[tag] = r + for model, runs in js.SECONDARY: + if args.primary_only: + continue + cl_s, c_s = gather( + args.study_dir, args.out_dir, [model], runs, arm, 0, + lenient=lenient, meetings=meetings, + ) # fmt: skip + r = _summarise(cl_s) + r["counts"] = c_s + out["secondary"].setdefault(arm, {}).setdefault(model, {})[tag] = r + for seed in SEEDS[1:]: + cl_x, _c = gather( + args.study_dir, args.out_dir, prim_models, (1,), arm, seed, + lenient=False, meetings=meetings, + ) # fmt: skip + out["stability"].setdefault(arm, {})[f"s{seed}"] = _summarise(cl_x) + verdicts: dict[str, str] = {} + for arm in ARMS: + row = out["primary"][arm]["strict"] + counts = row["counts"] + cut_p = ( + row["cut"]["supported"] / row["cut"]["labelled"] + if row["cut"]["labelled"] + else 0.0 + ) + if "delta" not in row: + verdicts[arm] = "insufficient" + else: + verdicts[arm] = classify(row["delta"], cut_p, row["cut"]["labelled"]) + if degraded(counts["windows_unlabelled"], counts["windows"]): + verdicts[arm] = "degraded" + labelled = [v for v in verdicts.values()] + out["decision"] = { + "arms": verdicts, + "verdict": "degraded" if "degraded" in labelled else overall(labelled), + } + path = args.out_dir / "scores" / "cut_precision.json" + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(out, indent=2) + "\n", encoding="utf-8") + print(render(out)) + print(f"wrote {path}") + return 0 + + +def _frac(d: Mapping[str, int]) -> str: + n, t = d["supported"], d["labelled"] + return f"{n} of {t} ({n / t:.3f})" if t else "0 of 0" + + +def _ci(row: Mapping[str, Any]) -> str: + d = row.get("delta") + return ( + "-" if d is None else f"{d['point']:+.4f} [{d['low']:+.4f}, {d['high']:+.4f}]" + ) + + +def render(out: Mapping[str, Any]) -> str: + lines: list[str] = [] + head = "| Scope | arm | kept precision | cut precision | kept - cut [95% CI] |" + sep = "|---|---|---|---|---|" + lines += ["\n## Primary, pooled (strict, then lenient)\n", head, sep] + for arm in ARMS: + for tag in ("strict", "lenient"): + r = out["primary"][arm][tag] + lines.append( + f"| all 5, {tag} | {arm} | {_frac(r['kept'])} | {_frac(r['cut'])} | {_ci(r)} |" + ) + lines += ["\n## Primary, per extractor (strict)\n", head, sep] + for arm in ARMS: + for model, tags in out["primary"][arm]["by_extractor"].items(): + r = tags["strict"] + lines.append( + f"| `{model}` | {arm} | {_frac(r['kept'])} | {_frac(r['cut'])} | {_ci(r)} |" + ) + if out["secondary"]: + lines += ["\n## Secondary, runs 1 and 2 (strict)\n", head, sep] + for arm, models in out["secondary"].items(): + for model, tags in models.items(): + r = tags["strict"] + lines.append( + f"| `{model}` | {arm} | {_frac(r['kept'])} | {_frac(r['cut'])} | {_ci(r)} |" + ) + lines += ["\n## Seed stability (strict, primary pooled)\n", head, sep] + for arm, seeds in out["stability"].items(): + for tag, r in seeds.items(): + lines.append( + f"| {tag} | {arm} | {_frac(r['kept'])} | {_frac(r['cut'])} | {_ci(r)} |" + ) + lines.append("\n## Coverage\n") + for arm in ARMS: + c = out["primary"][arm]["strict"]["counts"] + lines.append( + f"- {arm}: windows labelled {c['windows'] - c['windows_unlabelled']} of " + f"{c['windows']}; windowed objects {c['objects_windowed']}; " + f"no-window objects excluded {c['objects_no_window']}; " + f"records skipped {c['records_skipped']}" + ) + d = out["decision"] + lines.append(f"\narms: {d['arms']}; verdict: {d['verdict']}") + return "\n".join(lines) + + +# --- self-test ----------------------------------------------------------------- + + +def _check(cond: bool, message: str) -> None: + if not cond: + raise AssertionError(message) + + +def _o(title: str, window: int | None = 0) -> Obj: + return {"type": "Event", "title": title, "description": "d", "window": window} + + +def _t_partition() -> None: + ceiling = [_o("a"), _o("b"), _o("b"), _o("c", None)] + _check( + partition(ceiling, [_o("b"), _o("c", None)]) == [False, True, False, True], + "multiset match, first duplicate kept", + ) + _check(partition(ceiling, []) == [False] * 4, "nothing kept") + _check(partition([], []) == [], "empty") + _check(windowed(ceiling) == {0: [0, 1, 2]}, "windowless objects are excluded") + + +def _t_items() -> None: + flags = [True, False, False] + labels = {0: "supported", 1: "partial", 2: "unsupported"} + _check(window_item(labels, flags, lenient=False) == (1, 1, 0, 2), "strict") + _check(window_item(labels, flags, lenient=True) == (1, 1, 1, 2), "lenient") + + +def _t_rule() -> None: + def d(p: float, lo: float, hi: float) -> dict[str, float]: + return {"point": p, "low": lo, "high": hi} + + _check(classify(d(0.2, 0.08, 0.3), 0.7, 40) == "buys_precision", "buys") + _check(classify(d(0.0, -0.1, 0.04), 0.85, 40) == "too_strict", "too strict") + _check(classify(d(0.0, -0.1, 0.04), 0.79, 40) == "inconclusive", "cut floor") + _check(classify(d(0.05, -0.02, 0.12), 0.9, 40) == "inconclusive", "straddles") + _check(classify(d(0.0, -0.1, 0.04), 0.9, 29) == "insufficient", "min cut") + _check(classify(d(0.2, 0.05, 0.3), 0.7, 40) == "inconclusive", "bound is strict") + _check(overall(["too_strict", "too_strict"]) == "too_strict", "both strict") + _check(overall(["buys_precision", "buys_precision"]) == "buys_precision", "both") + _check(overall(["too_strict", "buys_precision"]) == "mixed_or_inconclusive", "mix") + _check(overall([]) == "mixed_or_inconclusive", "empty") + _check(degraded(2, 20) is False and degraded(3, 20) is True, "degraded") + + +def _t_summary() -> None: + clusters: dict[str, list[Item]] = { + "m1": [(3, 3, 1, 2), (2, 2, 0, 1)], + "m2": [(1, 2, 1, 1)], + } + row = _summarise(clusters) + _check(row["kept"] == {"supported": 6, "labelled": 7}, f"kept {row['kept']}") + _check(row["cut"] == {"supported": 2, "labelled": 4}, f"cut {row['cut']}") + want = 6 / 7 - 2 / 4 + _check(abs(row["delta"]["point"] - want) < 1e-12, "point is pooled difference") + _check(row["delta"]["low"] <= row["delta"]["point"] <= row["delta"]["high"], "ci") + + +def self_test() -> int: + tests: list[Callable[[], None]] = [_t_partition, _t_items, _t_rule, _t_summary] + failed = 0 + for test in tests: + try: + test() + print(f"ok {test.__name__}") + except AssertionError as exc: + failed += 1 + print(f"FAIL {test.__name__}: {exc}") + print(f"self-test: {len(tests) - failed} of {len(tests)} passed") + return 1 if failed else 0 + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument( + "--self-test", action="store_true", help="model-free checks, then exit" + ) + parser.add_argument("--screen-dir", type=pathlib.Path, default=SCREEN_DIR) + parser.add_argument("--study-dir", type=pathlib.Path, default=STUDY_DIR) + parser.add_argument("--out-dir", type=pathlib.Path, default=OUT_DIR) + parser.add_argument("--host", default=None) + parser.add_argument("--primary-only", action="store_true") + sub = parser.add_subparsers(dest="command") + lab = sub.add_parser("label") + lab.add_argument("--models", nargs="*", default=None) + sub.add_parser("score") + args = parser.parse_args(argv) + if args.self_test: + return self_test() + commands = {"label": cmd_label, "score": cmd_score} + if args.command not in commands: + raise SystemExit("choose a subcommand, or --self-test") + return commands[args.command](args) + + +if __name__ == "__main__": + raise SystemExit(main()) From 5be52e046604464051617d0d9e33ecf7f2ac53cf Mon Sep 17 00:00:00 2001 From: Jason Date: Tue, 20 Oct 2026 09:10:12 -0500 Subject: [PATCH 3/4] docs(evals): record the selector judge cut-precision result, inconclusive by the registered rule (#1579) --- .../CUT_PRECISION_RESULTS.md | 123 ++++++++++++++++++ evals/concept_model_screen/cut_precision.py | 6 +- 2 files changed, 128 insertions(+), 1 deletion(-) create mode 100644 evals/concept_model_screen/CUT_PRECISION_RESULTS.md diff --git a/evals/concept_model_screen/CUT_PRECISION_RESULTS.md b/evals/concept_model_screen/CUT_PRECISION_RESULTS.md new file mode 100644 index 00000000..d4275611 --- /dev/null +++ b/evals/concept_model_screen/CUT_PRECISION_RESULTS.md @@ -0,0 +1,123 @@ +# Results: does the selector judge's cut buy precision? (#1579, step 1) + +**Verdict by the registered rule: mixed or inconclusive. Both arms fall in +rule 3.** The data do not show that the cut buys precision, but they cannot +exclude a benefit of up to about 7 points. + +- Cut objects are as source-supported as kept ones at the point estimate: + - `self` judge: kept 153 of 165 (0.927), cut 128 of 138 (0.928). + - `fixed` judge: kept 147 of 160 (0.919), cut 134 of 143 (0.937). +- Rule 1 (the cut buys precision) fails clearly: the lower bound of kept minus + cut is -0.068 (`self`) and -0.096 (`fixed`), far under +0.05. +- Rule 2 (the cut is too strict) fails narrowly: the upper bound is +0.0689 + (`self`) and +0.0510 (`fixed`), against the registered +0.05. The cut + precision floor of 0.80 held in both arms (0.928, 0.937). + +The rules were registered in +[`CUT_PRECISION_PREREGISTRATION.md`](CUT_PRECISION_PREREGISTRATION.md), merged +before the first `gpt-oss:20b` call. This is step 1 of #1579. It changes no +production code and no default. + +## Run + +- The harness is `cut_precision.py` (`label`, `score`, `--self-test`). +- Only `gpt-oss:20b` (digest `f38aa0c53da5…`, the #1557 guardrail judge) was + called, with the #1557 G1 prompt and parser. No extraction, no selector-judge + call, no embedding. The #1557 and #1571 directories were read only; labels + are under `~/Dev/Projects/openkos-eval-runs/cut-precision/`. +- Ollama was used by this study alone. +- **216 of 216 window calls** (154 primary, 62 secondary): 0 unparsed. No window was excluded, so the run is + not degraded. +- Wall time: **about 36 min** (08:33 to 09:08; 2137 s in judge calls, 9.9 s + per call on average) against the forecast of about 30 min (upper estimate + 1 h). The population was not reduced. +- One label per ceiling object, in one call per window, so cut and kept are + judged together for both arms and every seed. +- Primary: 303 windowed ceiling objects (154 windows) of 352 ceiling objects; + 49 objects with no window were excluded as registered. 303 = 165 + 138 + (`self`) = 160 + 143 (`fixed`). +- Validity: 0 of 25 primary records skipped for a ceiling that differs from + the recorded one. `gemma4:26b-a4b` is one sample in both arms (its judge is + the fixed judge). + +## Primary: round 1, run 1, pool DA, five extractors pooled + +Strict precision = supported of labelled. Lenient = supported or partial of +labelled, reported only. Intervals: two-stage cluster bootstrap (meetings, +then windows; 10,000 resamples; seed 42) on the pooled difference +kept minus cut. + +| Arm | Precision | kept | cut | kept - cut [95% CI] | Registered label | +|---|---|---|---|---|---| +| `self` | strict | 153 of 165 (0.927) | 128 of 138 (0.928) | -0.0003 [-0.0681, +0.0689] | inconclusive | +| `self` | lenient | 164 of 165 (0.994) | 134 of 138 (0.971) | +0.0229 [-0.0225, +0.0799] | not decided | +| `fixed` | strict | 147 of 160 (0.919) | 134 of 143 (0.937) | -0.0183 [-0.0963, +0.0510] | inconclusive | +| `fixed` | lenient | 159 of 160 (0.994) | 139 of 143 (0.972) | +0.0217 [-0.0229, +0.0788] | not decided | + +## Per extractor (strict, reported only) + +| Extractor | Arm | kept | cut | kept - cut [95% CI] | +|---|---|---|---|---| +| `qwen3:8b` | self | 18 of 18 (1.000) | 4 of 4 (1.000) | +0.0000 [+0.0000, +1.0000] | +| `qwen3.6:35b-a3b` | self | 27 of 29 (0.931) | 24 of 26 (0.923) | +0.0080 [-0.1818, +0.1512] | +| `gemma4:26b-a4b` | self = fixed | 37 of 40 (0.925) | 25 of 29 (0.862) | +0.0629 [-0.0927, +0.2780] | +| `qwen3:14b` | self | 57 of 62 (0.919) | 39 of 42 (0.929) | -0.0092 [-0.1385, +0.1111] | +| `qwen3.6:27b` | self | 14 of 16 (0.875) | 36 of 37 (0.973) | -0.0980 [-0.3478, +0.0952] | +| `qwen3:8b` | fixed | 11 of 11 (1.000) | 11 of 11 (1.000) | +0.0000 [+0.0000, +0.0000] | +| `qwen3.6:35b-a3b` | fixed | 31 of 33 (0.939) | 20 of 22 (0.909) | +0.0303 [-0.1739, +0.2143] | +| `qwen3:14b` | fixed | 44 of 49 (0.898) | 52 of 55 (0.945) | -0.0475 [-0.2396, +0.0938] | +| `qwen3.6:27b` | fixed | 24 of 27 (0.889) | 26 of 26 (1.000) | -0.1111 [-0.2692, +0.0000] | + +The extractor-level cut sets are small (4 to 55 labelled objects). `qwen3:8b`, +the production extractor, has 4 cut objects under its own judge and 11 under +the fixed judge, so nothing is decided for it alone. + +## Secondary: runs 1 and 2 (strict, reported only) + +| Extractor | Arm | kept | cut | kept - cut [95% CI] | +|---|---|---|---|---| +| `qwen3:8b` | self | 40 of 41 (0.976) | 13 of 13 (1.000) | -0.0244 [-0.0968, +0.0000] | +| `qwen3:8b` | fixed | 29 of 29 (1.000) | 24 of 25 (0.960) | +0.0400 [+0.0000, +0.1429] | +| `qwen3.6:27b` | self | 28 of 31 (0.903) | 80 of 85 (0.941) | -0.0380 [-0.1678, +0.0745] | +| `qwen3.6:27b` | fixed | 51 of 56 (0.911) | 57 of 60 (0.950) | -0.0393 [-0.1613, +0.0962] | + +## Seed stability (strict, primary pooled) + +| Seed index | Arm | kept | cut | kept - cut [95% CI] | +|---|---|---|---|---| +| s1 | self | 154 of 164 (0.939) | 127 of 139 (0.914) | +0.0254 [-0.0512, +0.1156] | +| s2 | self | 155 of 169 (0.917) | 126 of 134 (0.940) | -0.0231 [-0.0916, +0.0484] | +| s1 | fixed | 132 of 143 (0.923) | 149 of 160 (0.931) | -0.0082 [-0.0785, +0.0583] | +| s2 | fixed | 137 of 152 (0.901) | 144 of 151 (0.954) | -0.0523 [-0.1356, +0.0249] | + +The difference moves between -0.052 and +0.025 across seed indices: the +judge's choice of which objects to cut is as noisy as the effect. + +## What this does and does not say + +- **No evidence that the cut buys precision.** In every arm and seed, the cut + objects are supported at 0.91 to 0.95 (strict) and 0.97 (lenient), the same + as the kept ones. At seed 0 the difference is -0.0003 and -0.018. +- **Not enough to call the cut too strict by the registered rule.** The + upper bounds sit at +0.069 and +0.051, not below +0.05. With 138 and 143 cut + objects, the interval is about 14 points wide. +- **Precision here is "supported by the window".** The ceiling is already + about 0.92 supported for kept and cut alike, so there is little headroom for + a precision-buying cut. The judge's other stated aim, dropping vague or + near-duplicate objects, is not what G1 measures: a duplicate of a supported + object is supported. A cut that removes duplicates would show here as + supported-and-cut, so this measurement cannot tell a harmless duplicate cut + from a wrongly cut distinct object. +- **Limits as registered.** One labeller, one label per object, labels from + the ceiling batch and not the final batch (absolute levels are not + comparable to #1557's 0.927 and 0.903), a window-only source. + +## Consequence for #1579 + +By the registered rule neither step-2 branch is established. The +measurement supports a narrower claim than "the cut is too strict": the cut +does not buy source-grounded precision at this n. Choosing between loosening +the selection and recording the lift cost as the price of precision needs a +measure the G1 label does not give (for example duplicate and vagueness +labels on the cut objects), or a larger n. No default changes, and any change +needs an ADR. diff --git a/evals/concept_model_screen/cut_precision.py b/evals/concept_model_screen/cut_precision.py index 03e0ac59..8d717ace 100644 --- a/evals/concept_model_screen/cut_precision.py +++ b/evals/concept_model_screen/cut_precision.py @@ -280,7 +280,11 @@ def gather( lab = _load(label_path(out_dir, model, meeting, run)) base = _load(js.replay_path(study_dir, "self", model, meeting, run, 0)) ceiling = base["stages"]["ceiling"] or [] - rec = _load(js.replay_path(study_dir, arm, model, meeting, run, seed)) + # gemma4:26b-a4b judges itself and is the fixed judge: one sample. + judge_arm = "fixed" if model == js.JUDGE_MODEL else arm + rec = _load( + js.replay_path(study_dir, judge_arm, model, meeting, run, seed) + ) if not js.same_objects(rec["stages"]["ceiling"], ceiling): counts["records_skipped"] += 1 continue From 552974b3491fbc62d3ae7c956ac7280158c4328d Mon Sep 17 00:00:00 2001 From: Jason Date: Tue, 20 Oct 2026 09:58:37 -0500 Subject: [PATCH 4/4] docs(evals): disclose the gemma4 self-arm change as a deviation from the pre-registration (#1579) The change was made after the labels existed. The pooled verdict is unchanged without that extractor (mixed or inconclusive), though the fixed arm alone reads rule 2. Refs #1579. --- .../CUT_PRECISION_RESULTS.md | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/evals/concept_model_screen/CUT_PRECISION_RESULTS.md b/evals/concept_model_screen/CUT_PRECISION_RESULTS.md index d4275611..671b9ec7 100644 --- a/evals/concept_model_screen/CUT_PRECISION_RESULTS.md +++ b/evals/concept_model_screen/CUT_PRECISION_RESULTS.md @@ -112,6 +112,45 @@ judge's choice of which objects to cut is as noisy as the effect. the ceiling batch and not the final batch (absolute levels are not comparable to #1557's 0.927 and 0.903), a window-only source. +## Deviations from the pre-registration + +Disclosed after the fact. The pre-registration was committed at `fc44c899`; +`cut_precision.py` did not exist then, and the diff of the harness and of +these results against that commit is the two files in full. The only change +to the harness between its own first commit (`692aba70`) and the results +commit (`ec73a85b`) is the one below. No other post-registration change was +found: the margin, floor, minimum cut, bootstrap settings and rules in the +code match the registered text. + +- **`gemma4:26b-a4b` reads its `fixed` records in the `self` arm.** + - What: in `gather`, an extractor equal to the fixed judge model takes its + kept/cut partition from the `fixed` replay records in both arms, instead + of from the `self` records. + - When: made after the labels had been seen (the `label` run had finished), in + the results commit, not before the first label. + - Why: `gemma4:26b-a4b` is both the extractor and the fixed judge, so its + `self` and `fixed` arms are the same sample. The pre-registration says + they are "reported once"; the harness as first committed did not enforce + that, and the fix makes the two arms share one sample for that extractor. + - Effect on the pooled verdict: none. Recomputed from the stored labels and + replay records with no model call (the same `gather` and bootstrap, + seed 42, over the four other extractors only), strict, primary, seed 0: + - `self`: kept 116 of 125 (0.928), cut 103 of 109 (0.945), kept - cut + -0.0170 [-0.0964, +0.0618]: rule 3, inconclusive (as registered). + - `fixed`: kept 110 of 120 (0.917), cut 109 of 114 (0.956), kept - cut + -0.0395 [-0.1336, +0.0334]: rule 2 on its own (the upper bound falls + under +0.05 and the cut precision is 0.956). + - Overall: one arm rule 2 and one rule 3, so **mixed or inconclusive**, + the same as the reported verdict. The `fixed` arm alone would flip from + inconclusive to "too strict" without `gemma4:26b-a4b`, so that arm's + label depends on whether this extractor is pooled. It is pooled as + registered ("all five extractors"). + - The pre-fix number was not computed: it would need the pre-fix path and + its inputs for `self` records of an extractor whose `self` arm is not a + distinct sample. +- The four-extractor figures above are a sensitivity check run from a + throwaway script, not a registered analysis and not stored by the harness. + ## Consequence for #1579 By the registered rule neither step-2 branch is established. The