diff --git a/site/.firebaserc b/site/.firebaserc index 1f8f65a9..adb750c7 100644 --- a/site/.firebaserc +++ b/site/.firebaserc @@ -1,5 +1,5 @@ { "projects": { - "default": "devops-bench-demo" + "default": "devops-bench-shared" } } diff --git a/site/README.md b/site/README.md index 5fcfba5f..7508cd37 100644 --- a/site/README.md +++ b/site/README.md @@ -293,15 +293,35 @@ Needs **Java** for the Firestore emulator ```bash # 1. start the emulator (Firestore :8080, UI :4000) -npx -y firebase-tools emulators:start --only firestore --project devops-bench-demo +npx -y firebase-tools emulators:start --only firestore --project devops-bench-shared # 2. in another shell: seed the emulator (writes to the leaderboard-test DB) -cd seed && FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-demo npm run seed && cd .. +cd seed && FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-shared npm run seed && cd .. # 3. run the app — firebase.js auto-connects to the emulator on localhost npm run dev # http://localhost:5173 ``` +> **The project id must match `VITE_FIREBASE_PROJECT_ID` in `.env`** (that is +> `devops-bench-shared`, the same project named at the top of this section). The +> emulator keeps each project id in a **separate namespace**, so seeding one id +> while the app reads another leaves the dashboard **silently empty** — no error, +> just no rows. Four places name the project and all four have to agree: +> +> - the `--project` flag in step 1 +> - `GCLOUD_PROJECT` in step 2 +> - `VITE_FIREBASE_PROJECT_ID` in `.env`, which is what the browser reads +> - `"default"` in `.firebaserc`, which is where the emulator falls back if you +> drop `--project` — get this one wrong and the emulator UI on :4000 shows no +> docs even though the seed reported success +> +> If that happens, query the emulator directly to see which namespace the data +> actually landed in: +> +> ```bash +> curl -s "http://127.0.0.1:8080/v1/projects/devops-bench-shared/databases/leaderboard-test/documents/setups" | head +> ``` + ### B) Staging (real cloud DB, fabricated data) > ✅ `leaderboard-test` is **already created and seeded** — just run the dev server: @@ -344,6 +364,7 @@ npm test # Vitest — fast, DB-free unit + component tests | `src/pages/Detail.test.jsx` | stat-card math (incl. null-safe / empty), task-table sorting, `?metric=` param, not-found/loading/error | | `src/hooks/useBenchmarkData.test.js` | load-once lifecycle, error capture, terminate-on-PROD | | `src/components/TrendChart.test.jsx` | sr-only a11y table: date-union columns, `—` for missing runs and null values | +| `src/components/MetricToggle.test.jsx` | a button per vocab metric, the quality/efficiency group split, `available` disabling | --- diff --git a/site/ingest/PROTOCOL.md b/site/ingest/PROTOCOL.md index 197b5ef4..0fad11bd 100644 --- a/site/ingest/PROTOCOL.md +++ b/site/ingest/PROTOCOL.md @@ -62,9 +62,17 @@ always `0`; the schema is already shaped for multi-iteration runs (§4). | `catastrophic` | boolean (optional) | `true` \| `false` | Whether a catastrophic tripwire fired (`cat_v = 0`), zeroing the outcome. Omitted by pre-v1 rows. | | `scoringVersion` | string (optional) | e.g. `"v1"` | Scoring-framework version that produced `outcomeScore`. Omitted by pre-v1 rows. | | `toolScore` | number \| null | `[0, 1]` or null | Tool-invocation score; `null` when unscored. | -| `latencySec` | number | `>= 0` | Agent wall-clock latency, seconds. | +| `latencySec` | number | `>= 0` | Agent wall-clock latency, seconds. Non-nullable, so the producer emits **`0` when latency was not measured**; derive reads `0` as that sentinel rather than as an instant run, and the `Latency` metric treats the row as missing data. | | `inputTokens` | integer \| null | `>= 0` or null | Prompt tokens consumed; `null` when usage was not captured. | | `outputTokens` | integer \| null | `>= 0` or null | Completion tokens produced; `null` when usage was not captured. | +| `cachedTokens` | integer \| null (optional) | `>= 0` or null | Cache-read input tokens. Omitted by pre-v1 rows. | +| `reasoningTokens` | integer \| null (optional) | `>= 0` or null | Reasoning/thinking tokens. A **sibling** bucket of `outputTokens`, not a subset, so the `Tokens` metric adds it in. Omitted by pre-v1 rows. | +| `cacheWriteTokens` | integer \| null (optional) | `>= 0` or null | Cache-creation input tokens. Omitted by pre-v1 rows. | +| `totalTokens` | integer \| null (optional) | `>= 0` or null | Provider-reported total. **Preferred over the bucket sum** when present, since it may count buckets the row does not break out. Omitted by pre-v1 rows. | + +The `Tokens` leaderboard metric is `totalTokens` when reported, else the sum of +whichever buckets are present, else `null` (no usage captured). A row reporting +neither a total nor any bucket is "not measured", not zero tokens. `setupId`, `model`, `harness`, `augmentation`, `runId`, `t` are **denormalized onto every row** — they repeat across all of a run's rows (they mirror the run's diff --git a/site/ingest/README.md b/site/ingest/README.md index 0fb397f7..5bf693fd 100644 --- a/site/ingest/README.md +++ b/site/ingest/README.md @@ -88,11 +88,11 @@ brew install openjdk export PATH="/opt/homebrew/opt/openjdk/bin:$PATH" # 2. Start the emulator (from site/, leave it running in another terminal): -cd .. && npx -y firebase-tools emulators:start --only firestore --project devops-bench-demo +cd .. && npx -y firebase-tools emulators:start --only firestore --project devops-bench-shared # Firestore on :8080, emulator UI on :4000 # 3. Ingest against it (from site/ingest/): -FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-demo \ +FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-shared \ node ingest.mjs fixtures/ ``` @@ -131,7 +131,7 @@ or `catalog.mjs` presentation? Re-score every setup from the **existing** raw rows — no re-upload (emulator from Option A still running, or ADC for real Firestore): ```bash -FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-demo \ +FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-shared \ node derive.mjs ``` diff --git a/site/ingest/derive.mjs b/site/ingest/derive.mjs index 9f3b98cd..d6290b36 100644 --- a/site/ingest/derive.mjs +++ b/site/ingest/derive.mjs @@ -12,10 +12,11 @@ // tasks <- distinct row.taskFolder at the LATEST run of each setup // history <- one aggregate point per distinct row.t, time-ordered // -// The SCORING FORMULA is NOT duplicated here — PASS_THRESHOLD and the pass@k -// estimator are imported from seed/mock-data.mjs so test data and real data are -// scored by exactly one definition. Change the formula there and re-run derive -// (see the CLI at the bottom) to re-score everything from the same raw rows. +// The SCORING FORMULA is NOT duplicated here — PASS_THRESHOLD, the pass@k +// estimator and the efficiency projection are imported from seed/mock-data.mjs +// so test data and real data are scored by exactly one definition. Change the +// formula there and re-run derive (see the CLI at the bottom) to re-score +// everything from the same raw rows. // // Presentation (order / color) is not derivable from results — it's curation, so // it comes from the optional catalog overrides, falling back to discovery order @@ -23,7 +24,7 @@ // catalog (see collectMetadata in catalog.mjs); this module only emits setups. // ============================================================================= -import { PASS_THRESHOLD, passAtK } from "../seed/mock-data.mjs"; +import { PASS_THRESHOLD, efficiencyFor, passAtK } from "../seed/mock-data.mjs"; import { PALETTE, SETUP_CATALOG } from "./catalog.mjs"; /** @@ -55,8 +56,16 @@ function round(v, dp) { function scoresFor(rows) { const scored = rows.filter(r => Number.isFinite(r.outcomeScore)); const n = scored.length; + // Efficiency is telemetry, not a score: it is recorded even for an iteration + // that never scored, so it is averaged over ALL rows rather than the scored + // subset, and it survives the no-scored-rows early return below. + const efficiency = efficiencyFor(rows); if (n === 0) { - return { pass1: null, pass5: null, passMax: null, composite: null, correctness: null, recoverableSafety: null }; + return { + pass1: null, pass5: null, passMax: null, + composite: null, correctness: null, recoverableSafety: null, + ...efficiency + }; } // pass1 thresholds on CORRECTNESS `c` (falling back to outcomeScore for // pre-v1 rows) so the pass rate isn't distorted by the √/gate composite. @@ -77,7 +86,8 @@ function scoresFor(rows) { passMax: null, composite: mean("outcomeScore"), correctness: mean("correctnessScore"), - recoverableSafety: mean("recoverableSafetyScore") + recoverableSafety: mean("recoverableSafetyScore"), + ...efficiency }; } @@ -95,7 +105,11 @@ function meanScores(scoreList) { passMax: avg("passMax"), composite: avg("composite"), correctness: avg("correctness"), - recoverableSafety: avg("recoverableSafety") + recoverableSafety: avg("recoverableSafety"), + latency: avg("latency"), + inputTokens: avg("inputTokens"), + outputTokens: avg("outputTokens"), + cachedTokens: avg("cachedTokens") }; } @@ -201,7 +215,7 @@ export function derive(rows, opts = {}) { // re-score every setup from the existing raw rows WITHOUT re-uploading. The // normal path (ingest.mjs) runs derive automatically after each upload. // -// FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-demo \ +// FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-shared \ // node derive.mjs // GCLOUD_PROJECT=devops-bench-shared FIRESTORE_DATABASE_ID=leaderboard-test \ // node derive.mjs diff --git a/site/ingest/derive.test.mjs b/site/ingest/derive.test.mjs index e8765d7d..896e7899 100644 --- a/site/ingest/derive.test.mjs +++ b/site/ingest/derive.test.mjs @@ -81,7 +81,84 @@ describe("derive — data-driven", () => { passMax: null, composite: null, correctness: null, - recoverableSafety: null + recoverableSafety: null, + // Efficiency is telemetry, not a score: an iteration that never + // scored still consumed wall-clock, so latency survives while every + // score is null. The token axes stay null because the fixture + // captured no usage. + latency: 1, + inputTokens: null, + outputTokens: null, + cachedTokens: null }); }); + + it("treats latencySec 0 as unmeasured, so it can't rank as the fastest", () => { + // Regression: latencySec is non-nullable upstream and normalize.py + // coerces a missing measurement to 0.0. Averaging that in gave an + // unmeasured setup a latency of 0 — first place on a lower-is-better + // metric, with a full bar. + const base = { + setupId: "s", model: "m", harness: "h", augmentation: [], + runId: "run_20260101_000000", t: "2026-01-01T00:00:00Z", + taskFolder: "task-a", taskName: "Task A", status: "success", + toolScore: null, inputTokens: null, outputTokens: null, outcomeScore: 0.9 + }; + const unmeasured = derive([{ ...base, iteration: 0, latencySec: 0 }]); + expect(unmeasured[0].tasks[0].scores.latency).toBeNull(); + + // A 0 alongside real readings drops out of the mean rather than halving it. + const mixed = derive([ + { ...base, iteration: 0, latencySec: 0 }, + { ...base, iteration: 1, latencySec: 10 } + ]); + expect(mixed[0].tasks[0].scores.latency).toBe(10); + }); + + it("projects the token buckets onto three axes rather than one total", () => { + // Regression: a single summed figure was ~the input count wearing a + // different label. Output was 0.7% of the fleet's summed buckets, so the + // most expensive axis was invisible in the number that ranked setups. + const base = { + setupId: "s", model: "m", harness: "h", augmentation: [], + runId: "run_20260101_000000", t: "2026-01-01T00:00:00Z", + taskFolder: "task-a", taskName: "Task A", status: "success", + toolScore: null, latencySec: 5, outcomeScore: 0.9, iteration: 0 + }; + const split = derive([ + { + ...base, + inputTokens: 100, + cacheWriteTokens: 50, + outputTokens: 200, + reasoningTokens: 700, + cachedTokens: 4000, + // A provider total no longer overrides the buckets: it cannot be + // attributed to an axis. + totalTokens: 950 + } + ]); + expect(split[0].tasks[0].scores).toMatchObject({ + inputTokens: 150, + outputTokens: 900, + cachedTokens: 4000 + }); + }); + + it("leaves the cached axis null for a harness that reports no cache reads", () => { + // Only some harnesses report cache reads. A blank cell has to stay + // distinct from a 0, or a silent harness would rank best on a + // lower-is-better axis purely for being less talkative. + const setups = derive([ + { + setupId: "s", model: "m", harness: "h", augmentation: [], + runId: "run_20260101_000000", t: "2026-01-01T00:00:00Z", + taskFolder: "task-a", taskName: "Task A", status: "success", + toolScore: null, latencySec: 5, outcomeScore: 0.9, iteration: 0, + inputTokens: 100, outputTokens: 200 + } + ]); + expect(setups[0].tasks[0].scores.cachedTokens).toBeNull(); + expect(setups[0].tasks[0].scores.inputTokens).toBe(100); + }); }); diff --git a/site/ingest/firestore.mjs b/site/ingest/firestore.mjs index 5bba2aeb..4a69e9e6 100644 --- a/site/ingest/firestore.mjs +++ b/site/ingest/firestore.mjs @@ -35,7 +35,7 @@ export function openDb() { const projectId = process.env.GCLOUD_PROJECT || process.env.GOOGLE_CLOUD_PROJECT || - (emulator ? "devops-bench-demo" : "devops-bench-shared"); + "devops-bench-shared"; const databaseId = process.env.FIRESTORE_DATABASE_ID || "leaderboard-test"; if (databaseId === PROD_DATABASE_ID && process.env.ALLOW_PROD_INGEST !== "true") { diff --git a/site/ingest/load.mjs b/site/ingest/load.mjs index 7051dee7..91cd59f5 100644 --- a/site/ingest/load.mjs +++ b/site/ingest/load.mjs @@ -83,9 +83,17 @@ export function validateRow(row) { if (!STATUSES.has(row.status)) errs.push('status: must be "success" or "failed"'); floatOrNull("outcomeScore", num01); floatOrNull("toolScore", num01); + // 0 is in contract: the producer's latencySec is non-nullable and coerces an + // unmeasured run to 0.0, which derive reads as the "not measured" sentinel (§2). float("latencySec", nonNeg); intOrNull("inputTokens", nonNeg); intOrNull("outputTokens", nonNeg); + // Extra usage buckets — OPTIONAL (pre-v1 rows omit them), validated when + // present because derive's token total reads them. A negative value here + // would drag the mean below zero and collapse every token bar to empty. + for (const k of ["cachedTokens", "reasoningTokens", "cacheWriteTokens", "totalTokens"]) { + if (k in row) intOrNull(k, nonNeg); + } // Scoring-framework v1 fields — OPTIONAL (pre-v1 rows omit them). Validate the // shape only when present so old runs still ingest. diff --git a/site/ingest/load.test.mjs b/site/ingest/load.test.mjs index 6deeb826..913d0352 100644 --- a/site/ingest/load.test.mjs +++ b/site/ingest/load.test.mjs @@ -82,6 +82,28 @@ describe("validateRow", () => { expect(errs.join()).toMatch(/inputTokens/); }); + it("accepts the extra usage buckets, present or absent", () => { + // Optional: pre-v1 rows omit them entirely and must still ingest. + expect(validateRow(validRow)).toEqual([]); + expect(validateRow({ + ...validRow, + cachedTokens: 100, reasoningTokens: 700, cacheWriteTokens: 0, totalTokens: 9520 + })).toEqual([]); + expect(validateRow({ + ...validRow, + cachedTokens: null, reasoningTokens: null, cacheWriteTokens: null, totalTokens: null + })).toEqual([]); + }); + + it("flags a negative or non-integer usage bucket", () => { + // Regression: a negative total reached the token mean and dragged + // the token mean below zero, collapsing every token bar to empty. + expect(validateRow({ ...validRow, totalTokens: -5 }).join()).toMatch(/totalTokens/); + expect(validateRow({ ...validRow, reasoningTokens: 1.5 }).join()).toMatch(/reasoningTokens/); + expect(validateRow({ ...validRow, cachedTokens: -1 }).join()).toMatch(/cachedTokens/); + expect(validateRow({ ...validRow, cacheWriteTokens: "1000" }).join()).toMatch(/cacheWriteTokens/); + }); + it("rejects a non-object", () => { expect(validateRow(null)).toEqual(["not an object"]); expect(validateRow([])).toEqual(["not an object"]); diff --git a/site/seed/mock-data.mjs b/site/seed/mock-data.mjs index e614e1ad..42266f4a 100644 --- a/site/seed/mock-data.mjs +++ b/site/seed/mock-data.mjs @@ -214,6 +214,14 @@ export function generateRaw() { latencySec: round(20 + rng() * 60, 2), inputTokens: Math.round(8000 + rng() * 30000), outputTokens: Math.round(300 + rng() * 1500), + // Cache reads are reported by SOME harnesses only (in the + // fleet's week-2 drop, 134 of 260 rows; in ours, none), so + // the mock leaves them absent on the API runner. That keeps + // a blank Cached cell on screen next to populated ones — + // the case the separate axis exists to make visible. + ...(def.harness === "api-loop" + ? {} + : { cachedTokens: Math.round(12000 + rng() * 60000) }), // Mock rows are all vetted so the seeded demo renders; // real rows carry per-task validated from the harness. validated: true @@ -260,8 +268,15 @@ function scoresFor(rows) { // denominator. With nothing scored there is no rate to report: null, not NaN. const scored = rows.filter(r => Number.isFinite(r.outcomeScore)); const n = scored.length; + // Efficiency is telemetry, not a score: averaged over ALL rows (an unscored + // iteration still consumed time and tokens) and survives the early return. + const efficiency = efficiencyFor(rows); if (n === 0) { - return { pass1: null, pass5: null, passMax: null, composite: null, correctness: null, recoverableSafety: null }; + return { + pass1: null, pass5: null, passMax: null, + composite: null, correctness: null, recoverableSafety: null, + ...efficiency + }; } // pass1 thresholds on CORRECTNESS `c` (falling back to outcomeScore for // pre-v1 rows), so the pass rate isn't distorted by the √/gate composite. @@ -282,7 +297,93 @@ function scoresFor(rows) { passMax: null, composite: mean("outcomeScore"), correctness: mean("correctnessScore"), - recoverableSafety: mean("recoverableSafetyScore") + recoverableSafety: mean("recoverableSafetyScore"), + ...efficiency + }; +} + +// --- efficiency projection --------------------------------------------------- +// +// Exported for the same reason PASS_THRESHOLD and passAtK are: ingest/derive.mjs +// projects efficiency from real rows and must use exactly this definition, not a +// copy of it. Change a rule here and both mock and real data follow. + +// Mean of a raw (already-absolute) per-row value — seconds, token counts. No +// ×100: these are not fractions, and the UI formats them by unit. +export function rawMean(rows, pick) { + const vals = rows.map(pick).filter(v => Number.isFinite(v)); + return vals.length ? round(vals.reduce((a, b) => a + b, 0) / vals.length, 1) : null; +} + +// Wall-clock seconds for one row, or null when latency was never measured. +// Unlike every token bucket, `latencySec` is NON-nullable upstream (row.py) and +// normalize.py writes `float(record.get("latency") or 0.0)`, so an unmeasured +// run arrives as 0 rather than null. Treat 0 as that sentinel: a real agent run +// never completes in 0.0s, and averaging it in would rank an unmeasured setup +// FIRST on a lower-is-better metric with a full bar. +export function latencyOf(row) { + return Number.isFinite(row.latencySec) && row.latencySec > 0 ? row.latencySec : null; +} + +// Sum of the named buckets for one row, or null when none of them was captured. +// A non-positive result is the same unmeasured sentinel `latencyOf` handles: a +// harness that produced no session log normalizes to zeros rather than nulls +// (antigravity's parser returns `{input: 0, output: 0, total: 0, cached: 0}` +// for an empty log, and normalize.py coerces those through as ints), and no +// real run costs 0 tokens. Reporting the 0 would rank that setup FIRST on a +// lower-is-better metric. +function bucketSum(row, keys) { + const parts = keys.map(k => row[k]).filter(v => Number.isFinite(v)); + if (!parts.length) return null; + const total = parts.reduce((a, b) => a + b, 0); + return total > 0 ? total : null; +} + +// Token usage is reported as three axes, not one number, because the buckets are +// not interchangeable: a provider bills generated tokens at several times the +// prompt rate and a cache read at a fraction of it. Summing them reports +// whichever bucket happens to be largest — in our own drops output is 0.7% +// (fleet week-2), 2.4% and 5.5% of the summed buckets, so a combined figure is +// the input count wearing a different label, and the axis a reader cares about +// is the one it hides. +// +// Grouping is by BILLED RATE, so each axis is a quantity of one kind of thing: +// +// input = inputTokens + cacheWriteTokens — cache creation is prompt content +// sent for this run, billed at roughly the input rate. +// output = outputTokens + reasoningTokens — reasoning is a SIBLING bucket of +// output, not a subset (see the canonical buckets in normalize.py), +// and both are generated at the output rate. Folding it in here is +// what keeps reasoning models from being undercounted. +// cached = cachedTokens — the cheapest bucket, and reported by only some +// harnesses (134 of 260 fleet rows; none of ours). Folded into input +// it would make a harness that reports cache reads look more +// expensive than one that stays silent, which is a ranking artifact +// of telemetry verbosity rather than a real cost difference. +// +// None of these consults `totalTokens`: a single provider-reported total cannot +// be attributed to an axis, and no row in any dataset we hold carries one +// without also carrying the buckets. A producer that ever reports only a total +// renders blank here rather than being silently mis-attributed. +export function inputTokensOf(row) { + return bucketSum(row, ["inputTokens", "cacheWriteTokens"]); +} + +export function outputTokensOf(row) { + return bucketSum(row, ["outputTokens", "reasoningTokens"]); +} + +export function cachedTokensOf(row) { + return bucketSum(row, ["cachedTokens"]); +} + +// The efficiency slice of Scores for one group of iteration rows. +export function efficiencyFor(rows) { + return { + latency: rawMean(rows, latencyOf), + inputTokens: rawMean(rows, inputTokensOf), + outputTokens: rawMean(rows, outputTokensOf), + cachedTokens: rawMean(rows, cachedTokensOf) }; } @@ -299,7 +400,11 @@ function meanScores(scoreList) { passMax: avg("passMax"), composite: avg("composite"), correctness: avg("correctness"), - recoverableSafety: avg("recoverableSafety") + recoverableSafety: avg("recoverableSafety"), + latency: avg("latency"), + inputTokens: avg("inputTokens"), + outputTokens: avg("outputTokens"), + cachedTokens: avg("cachedTokens") }; } diff --git a/site/seed/mock-data.test.mjs b/site/seed/mock-data.test.mjs index 581af6a2..f154f837 100644 --- a/site/seed/mock-data.test.mjs +++ b/site/seed/mock-data.test.mjs @@ -1,5 +1,84 @@ import { describe, it, expect } from "vitest"; -import { generateRaw, derive, passAtK, PASS_THRESHOLD } from "./mock-data.mjs"; +import { + generateRaw, + derive, + passAtK, + PASS_THRESHOLD, + inputTokensOf, + outputTokensOf, + cachedTokensOf, + latencyOf +} from "./mock-data.mjs"; + +describe("token axes", () => { + // The whole point of the split: these three must never be summed into one + // figure, so each has to pick up exactly its own billed-rate family. + const row = { + inputTokens: 100, + cacheWriteTokens: 7, + outputTokens: 20, + reasoningTokens: 5, + cachedTokens: 400 + }; + + it("groups the buckets by billed rate, keeping the axes disjoint", () => { + expect(inputTokensOf(row)).toBe(107); // input + cache write + expect(outputTokensOf(row)).toBe(25); // output + reasoning + expect(cachedTokensOf(row)).toBe(400); // cache reads alone + }); + + it("counts reasoning as output, since it is a sibling bucket and not a subset", () => { + // Dropping it would undercount every reasoning model on the one axis + // that is billed at the highest rate. + expect(outputTokensOf({ outputTokens: 20, reasoningTokens: 5 })).toBe(25); + expect(outputTokensOf({ outputTokens: 20 })).toBe(20); + }); + + it("ignores a provider total, which cannot be attributed to an axis", () => { + // sumTokens used to prefer totalTokens. A total says nothing about the + // input/output split, so a row carrying only one reads as unmeasured + // rather than being silently attributed to a single axis. + const totalOnly = { totalTokens: 993225 }; + expect(inputTokensOf(totalOnly)).toBeNull(); + expect(outputTokensOf(totalOnly)).toBeNull(); + expect(cachedTokensOf(totalOnly)).toBeNull(); + // ...and a total alongside buckets never overrides them. + expect(inputTokensOf({ totalTokens: 993225, inputTokens: 135329 })).toBe(135329); + }); + + it("keeps cache reads off the input axis", () => { + // Only some harnesses report cache reads (134 of 260 fleet rows, none of + // ours). Folding them into input would make a harness that reports them + // look more expensive than one that stays silent. + expect(inputTokensOf({ inputTokens: 100, cachedTokens: 400 })).toBe(100); + expect(cachedTokensOf({ inputTokens: 100 })).toBeNull(); + }); + + it("is null when the axis was never captured at all", () => { + expect(inputTokensOf({ inputTokens: null })).toBeNull(); + expect(outputTokensOf({})).toBeNull(); + expect(cachedTokensOf({})).toBeNull(); + }); + + // Regression: antigravity's parser returns {input: 0, output: 0, total: 0, + // cached: 0} for an empty session log (parsing.py) and normalize.py coerces + // those through as ints rather than nulls. Reporting a real 0 would rank + // that setup FIRST on a lower-is-better metric with a full bar — the same + // failure latencyOf() guards against. + it("treats an all-zero usage record as unmeasured, not as zero tokens", () => { + const zeroed = { inputTokens: 0, outputTokens: 0, cachedTokens: 0, totalTokens: 0 }; + expect(inputTokensOf(zeroed)).toBeNull(); + expect(outputTokensOf(zeroed)).toBeNull(); + expect(cachedTokensOf(zeroed)).toBeNull(); + expect(latencyOf({ latencySec: 0 })).toBeNull(); + }); + + it("keeps an axis that was captured when a sibling axis is zeroed", () => { + // A zeroed bucket is per-axis: output going quiet must not blank input. + expect(inputTokensOf({ inputTokens: 400, outputTokens: 0 })).toBe(400); + expect(outputTokensOf({ inputTokens: 400, outputTokens: 0 })).toBeNull(); + }); +}); describe("passAtK", () => { it("is 0 when there are no passes", () => { @@ -95,7 +174,7 @@ describe("derive", () => { const task = derive(blanked) .find(x => x.id === s.id) .tasks.find(t => t.folder === folder); - expect(task.scores).toEqual({ + expect(task.scores).toMatchObject({ pass1: null, pass5: null, passMax: null, @@ -103,5 +182,10 @@ describe("derive", () => { correctness: null, recoverableSafety: null }); + // Efficiency is telemetry, not a score: the blanked cell still consumed + // wall-clock and tokens, so those survive while every score is null. + expect(task.scores.latency).toBeGreaterThan(0); + expect(task.scores.inputTokens).toBeGreaterThan(0); + expect(task.scores.outputTokens).toBeGreaterThan(0); }); }); diff --git a/site/seed/seed.mjs b/site/seed/seed.mjs index b5a49bb6..1faa273d 100644 --- a/site/seed/seed.mjs +++ b/site/seed/seed.mjs @@ -11,7 +11,7 @@ // Two targets, selected by whether FIRESTORE_EMULATOR_HOST is set: // // EMULATOR (default, no credentials needed): -// FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-demo \ +// FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-shared \ // node seed.mjs # → DB leaderboard-test // // REAL Firestore (the shared TEST database — uses Application Default Creds; @@ -32,7 +32,7 @@ const EMULATOR = !!process.env.FIRESTORE_EMULATOR_HOST; const PROJECT_ID = process.env.GCLOUD_PROJECT || process.env.GOOGLE_CLOUD_PROJECT || - (EMULATOR ? "devops-bench-demo" : "devops-bench-shared"); + "devops-bench-shared"; // Named Firestore database (must match the client's VITE_FIRESTORE_DATABASE_ID). // Defaults to the test DB; never silently defaults to prod. const DATABASE_ID = process.env.FIRESTORE_DATABASE_ID || "leaderboard-test"; diff --git a/site/src/components/LeaderboardRow.jsx b/site/src/components/LeaderboardRow.jsx index 5dcc2489..62343e59 100644 --- a/site/src/components/LeaderboardRow.jsx +++ b/site/src/components/LeaderboardRow.jsx @@ -4,12 +4,25 @@ import { Link } from "react-router-dom"; import { SetupIdentity } from "./SetupIdentity.jsx"; import { setupScore, setupLabel } from "../lib/accessors.js"; +import { formatMetric, metricBarFraction, metricMeta } from "../lib/vocab.js"; -export function LeaderboardRow({ setup, models, harnesses, metric }) { +// `metricBest` is the best value for this metric across the visible rows — for +// absolute metrics (latency, the token axes) that is the SMALLEST, and the bar shows +// each row's ratio to it, since those metrics have no natural ceiling. Unused by +// percentage metrics. +export function LeaderboardRow({ setup, models, harnesses, metric, metricBest }) { const model = models[setup.model]; const harness = harnesses[setup.harness]; - const score = setupScore(setup, metric) ?? 0; + const score = setupScore(setup, metric); + const barPct = metricBarFraction(metric, score, metricBest) * 100; const to = `/setup/${encodeURIComponent(setup.id)}?metric=${encodeURIComponent(metric)}`; + // The badge sits immediately left of the figure, so it reads as annotating + // it — and what a catastrophic violation zeroes is the OUTCOME score, not + // the seconds or tokens a run consumed. Those readings are unaffected and + // still valid, so under an efficiency metric the badge would flag a number + // it has no bearing on. It stays on the quality columns, where the zeroing + // is what the reader is looking at. + const badgeable = metricMeta(metric).percentage; return ( - {/* Score progression meter — fixed-width badge slot (reserved on every row) - keeps the %, bar, and column start identical whether or not a badge shows. */} + {/* Score progression meter. On a quality metric the badge slot is a + fixed width reserved on EVERY row, so the figure, bar and column + start line up whether or not a given row carries a badge. An + efficiency metric drops the slot entirely rather than reserving + empty space — no row can badge there, so the columns still agree + with each other and the bar gets the width back. */}
- - {setup.catastrophicCount > 0 && ( - - ⚠ {setup.catastrophicCount} - - )} - + {badgeable && ( + + {setup.catastrophicCount > 0 && ( + + ⚠ {setup.catastrophicCount} + + )} + + )} - {score.toFixed(1)}% + {formatMetric(metric, score)}
-
+
diff --git a/site/src/components/LeaderboardRow.test.jsx b/site/src/components/LeaderboardRow.test.jsx new file mode 100644 index 00000000..70cd315b --- /dev/null +++ b/site/src/components/LeaderboardRow.test.jsx @@ -0,0 +1,73 @@ +import { describe, it, expect, afterEach } from "vitest"; +import { render, screen, cleanup } from "@testing-library/react"; +import { MemoryRouter } from "react-router-dom"; +import { LeaderboardRow } from "./LeaderboardRow.jsx"; + +const MODELS = { "alpha-pro": { name: "Alpha Pro", provider: "Acme", logo: "alpha" } }; +const HARNESSES = { + "gemini-cli": { name: "Gemini CLI", type: "cli", accent: "#0ea5e9", logo: "terminal" } +}; + +// Two catastrophic tasks, and readings on both metric families so the row has +// something to print whichever one is selected. +const SETUP = { + id: "alpha-pro-gemini-cli", + order: 0, + model: "alpha-pro", + harness: "gemini-cli", + augmentation: [], + color: "#3b82f6", + catastrophicCount: 2, + tasks: [{ folder: "a", name: "A", scores: { composite: 84, latency: 40, outputTokens: 900 } }], + history: [{ t: "2026-01-15T00:00:00Z", scores: { composite: 84, latency: 40, outputTokens: 900 } }] +}; + +function renderRow(metric, setup = SETUP) { + return render( + + + + ); +} + +describe("LeaderboardRow catastrophic badge", () => { + afterEach(cleanup); + + it("badges a quality metric, where the violation is what zeroed the figure", () => { + renderRow("composite"); + expect(screen.getByText("⚠ 2")).toBeInTheDocument(); + }); + + it("drops the badge on the efficiency metrics, which a violation does not zero", () => { + // A catastrophic violation zeroes the OUTCOME score. The seconds and + // tokens a run consumed are unaffected and still valid, so a badge + // beside them would flag a reading it has no bearing on. + for (const metric of ["latency", "inputTokens", "outputTokens", "cachedTokens"]) { + renderRow(metric); + expect(screen.queryByText("⚠ 2")).not.toBeInTheDocument(); + cleanup(); + } + }); + + it("still prints the efficiency figure it was hiding the badge next to", () => { + // Guard against "fixing" the badge by dropping the whole slot including + // the value: the number is the point of the column. + renderRow("latency"); + expect(screen.getByText("40.0s")).toBeInTheDocument(); + }); + + it("shows no badge for a clean setup on any metric", () => { + const clean = { ...SETUP, catastrophicCount: 0 }; + for (const metric of ["composite", "latency"]) { + renderRow(metric, clean); + expect(screen.queryByText(/⚠/)).not.toBeInTheDocument(); + cleanup(); + } + }); +}); diff --git a/site/src/components/MetricToggle.jsx b/site/src/components/MetricToggle.jsx index 5a1d8788..f7fd2ea2 100644 --- a/site/src/components/MetricToggle.jsx +++ b/site/src/components/MetricToggle.jsx @@ -1,38 +1,69 @@ -// Pass@1 / Pass@5 / Pass^5 segmented control, shared by the leaderboard header -// and the detail hero. `available` (optional) marks which metrics have data — -// the others render DISABLED rather than hidden, so the UI advertises that -// pass5/passMax will return once the harness produces multi-iteration runs. -// When omitted (or empty), every metric is enabled (back-compat). +// Metric segmented control, shared by the leaderboard header and the detail +// hero. `available` (optional) marks which metrics have data — the others render +// DISABLED rather than hidden, so the UI advertises the axis exists and says why +// it's empty. The reason is per-metric (see metricUnavailableReason): pass@k is +// waiting on multi-iteration runs, the efficiency axes on harness telemetry. +// When `available` is omitted (or empty), every metric is enabled (back-compat). +// +// The metrics are split into two pills, quality and efficiency, rather than one +// long strip: ten buttons overflow the leaderboard's score column, and the +// break should fall BETWEEN the two families rather than mid-strip. Wrapping the +// pills puts it there, and the seam doubles as the visual cue that the efficiency +// axes read the other way — lower is better, absolute units. +// +// Both levels wrap. The pill seam is the PREFERRED break, but the quality group +// alone ("Recoverable Safety" plus five others) still outgrows a narrow score +// column, and a nowrap pill cannot shrink below its content — it just overflows +// again. Letting buttons wrap inside a pill makes overflow impossible at any +// width; the group split only decides where the break lands first. -import { METRICS, METRIC_LABELS, metricDescription } from "../lib/vocab.js"; +import { METRICS, METRIC_LABELS, metricDescription, metricMeta, metricShortLabel, metricUnavailableReason } from "../lib/vocab.js"; + +// Quality metrics first, then efficiency, each preserving METRICS order. Empty +// groups are dropped so a vocab with only one family renders a single pill. +function metricGroups() { + const quality = METRICS.filter(m => metricMeta(m).percentage); + const efficiency = METRICS.filter(m => !metricMeta(m).percentage); + return [quality, efficiency].filter(g => g.length > 0); +} export function MetricToggle({ value, onChange, available }) { const hasFilter = Array.isArray(available) && available.length > 0; const isEnabled = m => !hasFilter || available.includes(m); return ( -
- {METRICS.map(m => { - const active = m === value; - const enabled = isEnabled(m); - const cls = active - ? "bg-white dark:bg-slate-700 text-slate-800 dark:text-slate-100 shadow-sm" - : enabled - ? "text-slate-600 dark:text-slate-300 hover:text-slate-800 dark:hover:text-slate-100" - : "text-slate-300 dark:text-slate-600"; - return ( - - ); - })} +
+ {metricGroups().map(group => ( +
+ {group.map(m => { + const active = m === value; + const enabled = isEnabled(m); + const cls = active + ? "bg-white dark:bg-slate-700 text-slate-800 dark:text-slate-100 shadow-sm" + : enabled + ? "text-slate-600 dark:text-slate-300 hover:text-slate-800 dark:hover:text-slate-100" + : "text-slate-300 dark:text-slate-600"; + return ( + + ); + })} +
+ ))}
); } diff --git a/site/src/components/MetricToggle.test.jsx b/site/src/components/MetricToggle.test.jsx new file mode 100644 index 00000000..97c62ed9 --- /dev/null +++ b/site/src/components/MetricToggle.test.jsx @@ -0,0 +1,92 @@ +import { describe, it, expect, vi } from "vitest"; +import { render, screen, fireEvent } from "@testing-library/react"; + +import { MetricToggle } from "./MetricToggle.jsx"; +import { METRICS, METRIC_LABELS, metricDescription, metricMeta, metricShortLabel } from "../lib/vocab.js"; + +const buttonFor = metric => screen.getByRole("button", { name: METRIC_LABELS[metric] }); + +// The pill a button sits in — the split that keeps eight metrics from +// overflowing the leaderboard's score column. +const groupOf = metric => buttonFor(metric).parentElement; + +describe("MetricToggle", () => { + it("renders a button for every metric in the vocab", () => { + render( {}} />); + // Guards the whole vocab, not a hardcoded list: adding a metric key + // without a toggle button would leave it unreachable in the UI. + expect(screen.getAllByRole("button")).toHaveLength(METRICS.length); + for (const m of METRICS) expect(buttonFor(m)).toBeInTheDocument(); + }); + + it("splits quality and efficiency metrics into separate groups", () => { + render( {}} />); + const quality = METRICS.filter(m => metricMeta(m).percentage); + const efficiency = METRICS.filter(m => !metricMeta(m).percentage); + + // Every member of a family shares one pill, and the two pills differ — + // so a wrap falls between the families rather than mid-strip. + expect(new Set(quality.map(groupOf)).size).toBe(1); + expect(new Set(efficiency.map(groupOf)).size).toBe(1); + expect(groupOf(quality[0])).not.toBe(groupOf(efficiency[0])); + }); + + it("keeps METRICS order within each group", () => { + render( {}} />); + const rendered = screen.getAllByRole("button").map(b => b.textContent); + const expected = [ + ...METRICS.filter(m => metricMeta(m).percentage), + ...METRICS.filter(m => !metricMeta(m).percentage) + ].map(metricShortLabel); + expect(rendered).toEqual(expected); + }); + + it("abbreviates the visible text but keeps the full accessible name", () => { + render( {}} />); + // Shortening is a fit concern, not a vocabulary change: a screen reader + // and every query below still address the metric by its real label. + const button = buttonFor("recoverableSafety"); + expect(button).toHaveTextContent("Rec. Safety"); + expect(button).toHaveAccessibleName("Recoverable Safety"); + }); + + it("marks the active metric and reports a click", () => { + const onChange = vi.fn(); + render(); + expect(buttonFor("latency")).toHaveAttribute("aria-pressed", "true"); + expect(buttonFor("composite")).toHaveAttribute("aria-pressed", "false"); + + fireEvent.click(buttonFor("outputTokens")); + expect(onChange).toHaveBeenCalledWith("outputTokens"); + }); + + it("disables metrics missing from `available` rather than hiding them", () => { + const onChange = vi.fn(); + render(); + + expect(buttonFor("pass5")).toBeDisabled(); + expect(buttonFor("latency")).toBeEnabled(); + fireEvent.click(buttonFor("pass5")); + expect(onChange).not.toHaveBeenCalled(); + }); + + it("enables every metric when `available` is omitted or empty", () => { + render( {}} available={[]} />); + for (const m of METRICS) expect(buttonFor(m)).toBeEnabled(); + }); + + it("explains a disabled metric by the reason it is actually missing", () => { + // One hardcoded sentence used to cover every disabled button, so a + // harness that reports no timings put "Available once multi-iteration + // runs land" under Latency, which has nothing to do with iterations. + render( {}} available={["composite"]} />); + expect(buttonFor("pass5")).toHaveAttribute("title", "Available once multi-iteration runs land"); + expect(buttonFor("latency")).toHaveAttribute("title", "Not reported by these runs"); + expect(buttonFor("outputTokens")).toHaveAttribute("title", "Not reported by these runs"); + }); + + it("describes an enabled metric instead of explaining its absence", () => { + render( {}} available={["composite", "latency"]} />); + expect(buttonFor("latency")).toHaveAttribute("title", metricDescription("latency")); + }); +}); diff --git a/site/src/components/TrendChart.jsx b/site/src/components/TrendChart.jsx index 7ce8c52d..a65d1b25 100644 --- a/site/src/components/TrendChart.jsx +++ b/site/src/components/TrendChart.jsx @@ -14,6 +14,7 @@ import { Legend } from "chart.js"; import { setupHistory, setupLabel, allRunDates, formatRunDate, yAxisBounds } from "../lib/accessors.js"; +import { formatMetric, metricMeta } from "../lib/vocab.js"; import { useIsDark } from "../hooks/useIsDark.js"; // Register Chart.js parts + apply the Inter styling once at module load. Colors @@ -74,7 +75,7 @@ export function TrendChart({ tooltip: { callbacks: { title: items => (items.length ? formatRunDate(items[0].parsed.x) : ""), - label: ctx => ` ${ctx.dataset.label}: ${ctx.parsed.y.toFixed(1)}%` + label: ctx => ` ${ctx.dataset.label}: ${formatMetric(metric, ctx.parsed.y)}` } } }, @@ -93,14 +94,23 @@ export function TrendChart({ max: yBounds.max, border: { display: false }, grid: { color: gridColor }, - ticks: { color: textColor, callback: value => value + "%", stepSize: 10, padding: 8 } + // stepSize only suits the 0..100 percentage axis. An absolute + // metric's range is unbounded (tokens run to tens of thousands), + // so forcing a step of 10 would ask Chart.js for thousands of + // ticks; let it choose, capped, and format each by unit. + ticks: { + color: textColor, + callback: value => formatMetric(metric, value), + ...(metricMeta(metric).percentage ? { stepSize: 10 } : { maxTicksLimit: 8 }), + padding: 8 + } } }, elements: { line: { tension: 0.35, borderWidth: 3 }, point: { radius: 3, hitRadius: 12, hoverRadius: 6, hoverBackgroundColor: pointHover, hoverBorderWidth: 3 } } - }), [dates, showLegend, yBounds, textColor, gridColor, pointHover]); + }), [dates, metric, showLegend, yBounds, textColor, gridColor, pointHover]); return (
@@ -122,7 +132,7 @@ export function TrendChart({ // Guard both a missing run AND a present run with a // null value for this metric (sparse real data). const v = setup.history.find(h => h.t === d)?.scores[metric]; - return {v == null ? "—" : v.toFixed(1) + "%"}; + return {formatMetric(metric, v)}; })} ))} diff --git a/site/src/lib/accessors.js b/site/src/lib/accessors.js index 79b441b1..e99b8d6f 100644 --- a/site/src/lib/accessors.js +++ b/site/src/lib/accessors.js @@ -7,7 +7,7 @@ // so it stays a pure, easily-tested function. // ============================================================================= -import { AUGMENTATIONS, augmentationLabel } from "./vocab.js"; +import { AUGMENTATIONS, augmentationLabel, metricMeta } from "./vocab.js"; /** * @typedef {import('./schema').Setup} Setup @@ -42,6 +42,25 @@ export function setupLabel(setup, models, harnesses) { return parts.join(" · "); } +// Alphabetical ordering over the identity a row actually displays — model, then +// harness, then the augmentation chips — rather than over setup.id, which is a +// slug and would sort "api-loop" above "Gemini CLI" on punctuation the reader +// cannot see. `numeric` keeps gpt-5-2 ahead of gpt-5-10 instead of ordering the +// version digits as text. +/** + * @param {Setup} a + * @param {Setup} b + * @param {ModelMap} models + * @param {HarnessMap} harnesses + * @returns {number} + */ +export function compareByName(a, b, models, harnesses) { + const cmp = (x, y) => x.localeCompare(y, undefined, { numeric: true, sensitivity: "base" }); + return cmp(models[a.model].name, models[b.model].name) + || cmp(harnesses[a.harness].name, harnesses[b.harness].name) + || cmp(a.augmentation.join(","), b.augmentation.join(",")); +} + // Secondary modifier chips — one per augmentation token, or a single neutral // "Baseline" chip when the augmentation array is empty. The harness type chip // is built separately at the call site (it needs the per-harness accent color). @@ -92,6 +111,25 @@ export function formatRunDate(t) { return new Date(t).toLocaleDateString("en-CA", { timeZone: "UTC", year: "numeric", month: "2-digit", day: "2-digit" }); } +// A round tick step for an absolute range: 1, 2, 2.5 or 5 times a power of ten, +// the smallest that keeps the axis under ~8 gridlines. The percentage branch can +// hardcode 10 because its range is always 0..100; an absolute range spans +// seconds to tens of thousands of tokens, so the step has to follow the data. +function niceStep(range) { + const magnitude = 10 ** Math.floor(Math.log10(range / 4)); + for (const m of [1, 2, 2.5, 5]) { + if (range / (magnitude * m) <= 8) return magnitude * m; + } + return magnitude * 10; +} + +// Round `value` to a multiple of `step` with `round` (Math.floor / Math.ceil). +// The trailing toFixed(6) drops float dust so a 0.1 step yields 24.3, not +// 24.299999999999997, which would print as a long decimal on the axis. +function snap(value, step, round) { + return Number((round(value / step) * step).toFixed(6)); +} + // Trend-chart y-axis [min, max] for the given setups + metric. Fits the plotted // scores instead of a fixed 60–100 window so low scorers aren't clipped off the // bottom: pad by 5, snap to tens (keeps the 10-step ticks clean), clamp to @@ -107,8 +145,25 @@ export function yAxisBounds(setupsList, metric) { .map(p => p.y) .filter(y => y != null); if (!ys.length) return { min: 0, max: 100 }; - const min = Math.max(0, Math.floor((Math.min(...ys) - 5) / 10) * 10); - const max = Math.min(100, Math.ceil((Math.max(...ys) + 5) / 10) * 10); + const lo = Math.min(...ys); + const hi = Math.max(...ys); + // Absolute metrics (latency, tokens) have no 100 ceiling — clamping them + // there would flatten every series onto the top gridline. Pad by a tenth of + // the range instead and let the axis follow the data. + if (!metricMeta(metric).percentage) { + const pad = Math.max((hi - lo) * 0.1, hi * 0.05, 1); + // Snap to a round step for the same reason the percentage branch snaps + // to tens: Chart.js labels the endpoints as well as the interior ticks, + // so a raw padded bound prints a stray "54.8s" hard against the "54.0s" + // gridline above it. + const step = niceStep(hi + pad - Math.max(0, lo - pad)); + return { + min: Math.max(0, snap(lo - pad, step, Math.floor)), + max: snap(hi + pad, step, Math.ceil) + }; + } + const min = Math.max(0, Math.floor((lo - 5) / 10) * 10); + const max = Math.min(100, Math.ceil((hi + 5) / 10) * 10); // Guard against a zero-height axis when all points sit in one 10-wide band. return { min, max: max > min ? max : Math.min(100, min + 10) }; } diff --git a/site/src/lib/accessors.test.js b/site/src/lib/accessors.test.js index 8431989c..b5e82fd1 100644 --- a/site/src/lib/accessors.test.js +++ b/site/src/lib/accessors.test.js @@ -128,4 +128,46 @@ describe("yAxisBounds", () => { const allNull = makeSetup({ history: [{ t: "2026-01-15T00:00:00Z", scores: { pass1: null } }] }); expect(yAxisBounds([allNull], "pass1")).toEqual({ min: 0, max: 100 }); }); + + describe("absolute metrics", () => { + const withHistory = (metric, values) => makeSetup({ + history: values.map((v, i) => ({ + t: `2026-0${i + 1}-15T00:00:00Z`, + scores: { [metric]: v } + })) + }); + + it("follows the data instead of clamping to 100", () => { + // Token means run to five figures; a [0, 100] clamp would push every + // series off the top of the chart. + const b = yAxisBounds([withHistory("outputTokens", [23200, 24100, 25200])], "outputTokens"); + expect(b.min).toBeGreaterThan(100); + expect(b.min).toBeLessThan(23200); + expect(b.max).toBeGreaterThan(25200); + }); + + it("snaps the endpoints to a round step so no stray label crowds a gridline", () => { + // Raw padding gave [45.3, 54.7], printing "54.8s" against "54.0s". + expect(yAxisBounds([withHistory("latency", [47.9, 50.1, 52.2])], "latency")) + .toEqual({ min: 44, max: 56 }); + expect(yAxisBounds([withHistory("outputTokens", [23200, 24100, 25200])], "outputTokens")) + .toEqual({ min: 21000, max: 27000 }); + }); + + it("scales the step to the magnitude rather than assuming tens", () => { + // Sub-minute latencies need a fractional step; tokens need thousands. + const small = yAxisBounds([withHistory("latency", [2.1, 2.4])], "latency"); + expect(small).toEqual({ min: 1, max: 3.5 }); + }); + + it("keeps a non-zero height for a single data point", () => { + const b = yAxisBounds([withHistory("latency", [50])], "latency"); + expect(b.max).toBeGreaterThan(b.min); + }); + + it("never drops the axis below zero", () => { + const b = yAxisBounds([withHistory("latency", [0.5])], "latency"); + expect(b.min).toBeGreaterThanOrEqual(0); + }); + }); }); diff --git a/site/src/lib/schema.d.ts b/site/src/lib/schema.d.ts index 726d32b9..e4118d36 100644 --- a/site/src/lib/schema.d.ts +++ b/site/src/lib/schema.d.ts @@ -26,6 +26,12 @@ * The scoring metrics, in display order. `composite` is the scoring-framework v1 * headline (cat_v · √(c · rec_v)); `correctness` / `recoverableSafety` are its * sub-scores. All are 0..100 means. `pass1/5/Max` are pass rates. + * + * The efficiency keys are NOT percentages: they are absolute per-task means in + * seconds and token counts, where lower is better (see METRIC_META in vocab.js). + * The three token axes are separate because their buckets are billed at + * different rates and cannot be meaningfully added — see the grouping rules on + * `inputTokensOf` in seed/mock-data.mjs. */ export type MetricKey = | "composite" @@ -33,13 +39,19 @@ export type MetricKey = | "recoverableSafety" | "pass1" | "pass5" - | "passMax"; + | "passMax" + | "latency" + | "inputTokens" + | "outputTokens" + | "cachedTokens"; /** - * Per-metric scores as percentages (0..100). `null` where a metric has no - * scored data for the task/run. `pass5` and `passMax` are null today — they + * Per-metric values, keyed by MetricKey. Quality metrics are percentages + * (0..100); efficiency metrics are absolute magnitudes. `null` where a metric + * has no data for the task/run. `pass5` and `passMax` are null today — they * stay null until the harness produces multi-iteration runs (then `derive()` - * recomputes them from the same raw rows). + * recomputes them from the same raw rows). `cachedTokens` is null for any + * harness that does not report cache reads. */ export type Scores = Record; @@ -149,11 +161,23 @@ export interface ResultRow { scoringVersion?: string; /** Tool-use score in [0,1]; null when unscored. */ toolScore: number | null; + /** + * Agent wall-clock seconds. Non-nullable upstream, so an unmeasured run + * arrives as `0` — derive treats that as missing data, not an instant run. + */ latencySec: number; /** Null when token usage was not captured. */ inputTokens: number | null; /** Null when token usage was not captured. */ outputTokens: number | null; + /** Cache-read input tokens; null/absent when not captured. */ + cachedTokens?: number | null; + /** Reasoning tokens — a sibling bucket of `outputTokens`, not a subset. */ + reasoningTokens?: number | null; + /** Cache-creation input tokens; null/absent when not captured. */ + cacheWriteTokens?: number | null; + /** Provider-reported total; preferred over summing the buckets when present. */ + totalTokens?: number | null; /** Whether the task is vetted as correct; only validated tasks promote to the leaderboard. */ validated: boolean; } diff --git a/site/src/lib/vocab.js b/site/src/lib/vocab.js index 969b84c0..be2560de 100644 --- a/site/src/lib/vocab.js +++ b/site/src/lib/vocab.js @@ -33,12 +33,138 @@ export const METRIC_LABELS = { recoverableSafety: "Recoverable Safety", pass1: "Pass@1", pass5: "Pass@5", - passMax: "Pass^5" + passMax: "Pass^5", + latency: "Latency", + inputTokens: "Input Tokens", + outputTokens: "Output Tokens", + cachedTokens: "Cached Tokens" }; +// Abbreviated labels for the metric toggle only, where ten buttons compete for +// the width of one table column. "Recoverable Safety" is ~2.5x the width of any +// other button, so it alone decides whether the group fits on one line. Headings, +// tooltips and the accessible name of the button all keep the full METRIC_LABELS +// text — this shortens the visible glyphs, not the vocabulary. +// +// The token axes drop the shared "Tokens" noun because the efficiency pill +// already groups them next to each other: three buttons reading "Input Tokens / +// Output Tokens / Cached Tokens" repeat the word twice for no discrimination, +// and the full name survives in the tooltip and the accessible name. +const METRIC_SHORT_LABELS = { + recoverableSafety: "Rec. Safety", + inputTokens: "Input", + outputTokens: "Output", + cachedTokens: "Cached" +}; + +/** Toggle-button text for a metric, falling back to the full label. */ +export function metricShortLabel(metric) { + return METRIC_SHORT_LABELS[metric] ?? METRIC_LABELS[metric] ?? metric; +} + // The metric keys in display order — used by the metric toggles. Composite leads -// as the default headline; pass@k follow. -export const METRICS = ["composite", "correctness", "recoverableSafety", "pass1", "pass5", "passMax"]; +// as the default headline; pass@k follow, then the efficiency axes. +export const METRICS = [ + "composite", + "correctness", + "recoverableSafety", + "pass1", + "pass5", + "passMax", + "latency", + "inputTokens", + "outputTokens", + "cachedTokens" +]; + +// Per-metric presentation rules. Quality metrics are 0..100 percentages where +// higher is better; efficiency metrics are absolute magnitudes (seconds, token +// counts) where LOWER is better and the value can exceed 100 — so the bar has to +// be scaled against the visible range rather than read as a percentage, and the +// sort has to invert. Anything not listed defaults to the percentage rules. +const PERCENT = { unit: "%", lowerIsBetter: false, percentage: true }; +const TOKENS = { unit: "", lowerIsBetter: true, percentage: false }; +export const METRIC_META = { + composite: PERCENT, + correctness: PERCENT, + recoverableSafety: PERCENT, + pass1: PERCENT, + pass5: PERCENT, + passMax: PERCENT, + latency: { unit: "s", lowerIsBetter: true, percentage: false }, + inputTokens: TOKENS, + outputTokens: TOKENS, + cachedTokens: TOKENS +}; + +/** Presentation rules for a metric, defaulting to the percentage rules. */ +export function metricMeta(metric) { + return METRIC_META[metric] ?? PERCENT; +} + +/** True when a smaller value ranks better (latency, tokens). */ +export function isLowerBetter(metric) { + return metricMeta(metric).lowerIsBetter; +} + +/** + * Render a metric value for display: "85.4%", "42.1s", "12.3k". + * Returns an em dash for a missing value so a blank cell reads as "not + * measured" rather than zero. + */ +export function formatMetric(metric, value) { + if (value == null || !Number.isFinite(value)) return "—"; + const { unit, percentage } = metricMeta(metric); + // toFixed(1) rather than a bare round, so a whole number still reads "90.0%" + // and the column keeps a stable width across rows. + if (percentage) return `${value.toFixed(1)}%`; + if (unit === "s") return `${value.toFixed(1)}s`; + // Bare counts get thousands-compacted; a leaderboard cell has no room for + // "38412.0" and the exact figure is not what a reader is comparing. + if (value >= 1000) return `${(value / 1000).toFixed(1)}k`; + return String(Math.round(value)); +} + +/** + * Fraction (0..1) of the bar to fill for `value`. + * + * A percentage metric maps directly. An absolute metric has no natural ceiling, + * so it is expressed as a RATIO TO THE BEST value currently on screen (`best` = + * the smallest, since lower is better): the fastest/cheapest setup earns a full + * bar, and something twice as slow earns half of one. That keeps the "longer bar + * is better" reading every other metric has, while staying proportional — + * normalizing across `min..max` instead would render a 99s setup full and a 100s + * setup empty, exaggerating a 1% gap into the whole width. + */ +export function metricBarFraction(metric, value, best) { + if (value == null || !Number.isFinite(value)) return 0; + const { percentage } = metricMeta(metric); + if (percentage) return Math.max(0, Math.min(1, value / 100)); + // `value <= 0` can't be a real reading (0 latency is the unmeasured sentinel, + // 0 tokens means nothing was captured), and a non-positive best gives no scale. + if (!Number.isFinite(best) || best <= 0 || value <= 0) return 0; + return Math.max(0, Math.min(1, best / value)); +} + +/** + * The best value among `values` for `metric`, or null when none qualifies. + * + * "Best" follows the metric's direction: the smallest for a lower-is-better + * absolute metric, the largest otherwise. Non-positive readings are dropped + * from the lower-is-better case because they are the unmeasured sentinel + * (see `latencyOf` / `bucketSum` in seed/mock-data.mjs). That matters beyond + * the offending row: `best` is shared by the whole column, so one 0 would trip + * the `best <= 0` guard above and flatten EVERY bar, not just its own. + * + * Shared by the leaderboard (across visible setups) and the detail task table + * (across one setup's tasks) so the two cannot drift apart. + */ +export function bestValue(metric, values) { + const vals = values.filter(v => v != null && Number.isFinite(v)); + if (!isLowerBetter(metric)) return vals.length ? Math.max(...vals) : null; + const positive = vals.filter(v => v > 0); + return positive.length ? Math.min(...positive) : null; +} // One-line explanation per metric — the single source of truth for the score // tooltip (contextual to the selected metric) and each toggle button's hover. @@ -52,7 +178,14 @@ export const METRIC_DESCRIPTIONS = { pass1: "Pass@1: share of task attempts whose correctness clears the pass threshold (0.7).", pass5: "Pass@5: needs multi-iteration runs (not produced yet).", - passMax: "Pass^5: needs multi-iteration runs (not produced yet)." + passMax: "Pass^5: needs multi-iteration runs (not produced yet).", + latency: "Latency: mean agent wall-clock seconds per task. Lower is better, so the bar is scaled against the fastest setup on screen — a full bar is the fastest, half a bar is twice as slow.", + inputTokens: + "Input tokens: mean prompt tokens sent per task, including cache writes. Kept separate from output because providers bill it at a fraction of the generated rate. Lower is better.", + outputTokens: + "Output tokens: mean generated tokens per task, including reasoning tokens (a sibling of output, not a subset). The most expensive axis, and typically a few percent of the volume. Lower is better.", + cachedTokens: + "Cached tokens: mean cache-read tokens per task — prompt content billed at a steep discount. Reported by some harnesses only, so a blank cell means not reported, not zero. Lower is better." }; // Description for a metric key, falling back to its label. @@ -60,6 +193,27 @@ export function metricDescription(metric) { return METRIC_DESCRIPTIONS[metric] ?? METRIC_LABELS[metric] ?? metric; } +// Why a metric has no data, shown on its disabled toggle button. A metric can +// go missing for unrelated reasons — pass@k needs the same task run repeatedly, +// while an efficiency axis just needs the harness to report telemetry — so one +// hardcoded sentence can't cover both. It used to, which put "Available once +// multi-iteration runs land" under a greyed-out Latency button. +const METRIC_UNAVAILABLE_REASONS = { + pass5: "Available once multi-iteration runs land", + passMax: "Available once multi-iteration runs land", + latency: "Not reported by these runs", + inputTokens: "Not reported by these runs", + outputTokens: "Not reported by these runs", + // Cache reads are the one axis a harness can legitimately omit while + // reporting everything else, so the reason names the harness, not the run. + cachedTokens: "Not reported by these harnesses" +}; + +/** Tooltip for a metric with no data in the current dataset. */ +export function metricUnavailableReason(metric) { + return METRIC_UNAVAILABLE_REASONS[metric] ?? "Not reported by these runs"; +} + // Which metrics actually have any non-null value across the given setups. Used // by the metric toggle so pass@k buttons stay hidden until the harness // produces the multi-iteration runs that populate them. diff --git a/site/src/lib/vocab.test.js b/site/src/lib/vocab.test.js new file mode 100644 index 00000000..bc05082e --- /dev/null +++ b/site/src/lib/vocab.test.js @@ -0,0 +1,132 @@ +import { describe, it, expect } from "vitest"; +import { + formatMetric, + isLowerBetter, + metricBarFraction, + metricMeta, + bestValue, + METRICS, + METRIC_LABELS, + metricDescription +} from "./vocab.js"; + +describe("metric presentation rules", () => { + it("treats quality metrics as higher-is-better percentages", () => { + for (const m of ["composite", "correctness", "recoverableSafety", "pass1"]) { + expect(metricMeta(m).percentage).toBe(true); + expect(isLowerBetter(m)).toBe(false); + } + }); + + it("treats efficiency metrics as lower-is-better magnitudes", () => { + for (const m of ["latency", "inputTokens", "outputTokens", "cachedTokens"]) { + expect(metricMeta(m).percentage).toBe(false); + expect(isLowerBetter(m)).toBe(true); + } + }); + + it("keeps the token buckets on separate axes, never summed into one", () => { + // The buckets are billed at different rates, so a single combined + // metric reports whichever happens to be largest. Guard the vocabulary + // against a combined key creeping back in. + expect(METRICS).toEqual(expect.arrayContaining(["inputTokens", "outputTokens", "cachedTokens"])); + expect(METRICS).not.toContain("tokens"); + // Each carries its own label and its own explanation. + const labels = ["inputTokens", "outputTokens", "cachedTokens"].map(m => METRIC_LABELS[m]); + expect(new Set(labels).size).toBe(3); + const notes = ["inputTokens", "outputTokens", "cachedTokens"].map(metricDescription); + expect(new Set(notes).size).toBe(3); + }); + + it("defaults an unknown metric to the percentage rules", () => { + expect(metricMeta("nope").percentage).toBe(true); + }); +}); + +describe("formatMetric", () => { + it("keeps one decimal on percentages so the column width is stable", () => { + expect(formatMetric("composite", 90)).toBe("90.0%"); + expect(formatMetric("composite", 85.44)).toBe("85.4%"); + }); + + it("renders latency in seconds and compacts large token counts", () => { + expect(formatMetric("latency", 42.66)).toBe("42.7s"); + expect(formatMetric("latency", 8)).toBe("8.0s"); + expect(formatMetric("outputTokens", 38412)).toBe("38.4k"); + expect(formatMetric("outputTokens", 850)).toBe("850"); + }); + + it("renders a missing value as an em dash, never as zero", () => { + expect(formatMetric("latency", null)).toBe("—"); + expect(formatMetric("composite", undefined)).toBe("—"); + expect(formatMetric("outputTokens", NaN)).toBe("—"); + }); +}); + +describe("metricBarFraction", () => { + it("maps a percentage straight onto the bar", () => { + expect(metricBarFraction("composite", 75, null)).toBeCloseTo(0.75); + }); + + it("scales an absolute metric as a ratio to the best value on screen", () => { + // Best (fastest) is 10s: it earns a full bar, and twice as slow is half. + expect(metricBarFraction("latency", 10, 10)).toBeCloseTo(1); + expect(metricBarFraction("latency", 20, 10)).toBeCloseTo(0.5); + expect(metricBarFraction("latency", 100, 10)).toBeCloseTo(0.1); + }); + + it("gives the only visible setup a full bar, not a sliver", () => { + // Regression: filtering down to one row made value === the scale, which + // previously floored the bar at 2% for the fastest setup on screen. + expect(metricBarFraction("latency", 42, 42)).toBeCloseTo(1); + expect(metricBarFraction("outputTokens", 38412, 38412)).toBeCloseTo(1); + }); + + it("keeps near-equal values near-equal instead of full vs empty", () => { + // Regression: min..max normalization would render these 1.0 and 0.0, + // turning a 1% gap into the whole bar width. + expect(metricBarFraction("latency", 99, 99)).toBeCloseTo(1); + expect(metricBarFraction("latency", 100, 99)).toBeCloseTo(0.99, 2); + }); + + it("is empty for a missing value or an unusable scale", () => { + expect(metricBarFraction("latency", null, 100)).toBe(0); + expect(metricBarFraction("latency", 10, null)).toBe(0); + // 0 is the unmeasured sentinel, not an instant run — no bar for it. + expect(metricBarFraction("latency", 0, 10)).toBe(0); + expect(metricBarFraction("latency", 10, 0)).toBe(0); + }); +}); + +describe("bestValue", () => { + it("takes the smallest for a lower-is-better metric and the largest otherwise", () => { + expect(bestValue("latency", [30, 10, 20])).toBe(10); + expect(bestValue("outputTokens", [300, 100, 200])).toBe(100); + expect(bestValue("composite", [30, 10, 20])).toBe(30); + }); + + it("skips nulls and non-finite entries", () => { + expect(bestValue("latency", [null, 30, undefined, NaN, 20])).toBe(20); + expect(bestValue("latency", [null, undefined])).toBeNull(); + expect(bestValue("latency", [])).toBeNull(); + }); + + // Regression: an unmeasured run normalizes to 0 rather than null, and the + // min would then be 0. metricBarFraction bails on `best <= 0`, so that one + // row would empty EVERY bar in the column instead of only its own. + it("ignores the 0 sentinel so one unmeasured row cannot flatten the column", () => { + expect(bestValue("outputTokens", [0, 5000, 20000])).toBe(5000); + expect(bestValue("latency", [0, 42])).toBe(42); + expect(metricBarFraction("outputTokens", 20000, bestValue("outputTokens", [0, 5000, 20000]))).toBeCloseTo(0.25); + }); + + it("is null when every lower-is-better reading is the sentinel", () => { + expect(bestValue("outputTokens", [0, 0])).toBeNull(); + }); + + // A percentage metric legitimately bottoms out at 0 (a 0% pass rate), so the + // non-positive filter must not apply on that side. + it("keeps a genuine 0 for a higher-is-better metric", () => { + expect(bestValue("composite", [0, 0])).toBe(0); + }); +}); diff --git a/site/src/pages/Detail.jsx b/site/src/pages/Detail.jsx index 8176a45e..035bb6ed 100644 --- a/site/src/pages/Detail.jsx +++ b/site/src/pages/Detail.jsx @@ -6,7 +6,7 @@ import { useEffect, useMemo, useState } from "react"; import { useParams, useSearchParams, Link } from "react-router-dom"; import { useBenchmark } from "../context/BenchmarkContext.jsx"; import { setupScore, setupLabel } from "../lib/accessors.js"; -import { METRIC_LABELS, availableMetrics } from "../lib/vocab.js"; +import { METRICS, METRIC_LABELS, availableMetrics, formatMetric, metricBarFraction, isLowerBetter, metricMeta, bestValue } from "../lib/vocab.js"; import { SetupIdentity } from "../components/SetupIdentity.jsx"; import { MetricToggle } from "../components/MetricToggle.jsx"; import { TrendChart } from "../components/TrendChart.jsx"; @@ -31,24 +31,50 @@ function StatCard({ label, value, sub }) { function TaskTable({ setup, metric }) { const [sort, setSort] = useState({ key: "score", dir: "desc" }); + // Mirrors the leaderboard's ordering (Leaderboard.jsx): "desc" means BEST + // first, which for latency/tokens is the smallest value, and a task with no + // value for this metric sorts last in either direction rather than being + // read as a 0 — otherwise an unmeasured task would head the ascending list + // as if it were the fastest. const tasks = useMemo(() => { const dir = sort.dir === "asc" ? 1 : -1; - return [...setup.tasks].sort((a, b) => - sort.key === "name" - ? dir * a.name.localeCompare(b.name) - : dir * ((a.scores[metric] ?? 0) - (b.scores[metric] ?? 0)) - ); + const lower = isLowerBetter(metric); + return [...setup.tasks].sort((a, b) => { + if (sort.key === "name") return dir * a.name.localeCompare(b.name); + const av = a.scores[metric]; + const bv = b.scores[metric]; + if (av == null && bv == null) return 0; + if (av == null) return 1; + if (bv == null) return -1; + return dir * (lower ? bv - av : av - bv); + }); }, [setup, metric, sort]); + // Best value across this setup's tasks, so an absolute metric's bar has a + // scale (percentage metrics ignore it). Same helper the leaderboard uses, + // so the two cannot disagree about what "best" means. + const taskBest = useMemo( + () => bestValue(metric, setup.tasks.map(t => t.scores[metric])), + [setup, metric] + ); + function sortBy(key) { setSort(prev => prev.key === key ? { key, dir: prev.dir === "asc" ? "desc" : "asc" } : { key, dir: key === "name" ? "asc" : "desc" }); } - const Arrow = ({ k }) => sort.key === k - ? {sort.dir === "asc" ? "▲" : "▼"} - : ↕; + // The arrow reports which way the VALUES run, not the internal sort flag. + // "desc" means best-first, and best-first under latency/tokens is ascending + // numbers — so the glyph has to invert or the column reads 22.3k → 28.0k + // under a ▼. + const Arrow = ({ k }) => { + if (sort.key !== k) return ↕; + const ascending = k === "name" + ? sort.dir === "asc" + : (sort.dir === "asc") !== isLowerBetter(metric); + return {ascending ? "▲" : "▼"}; + }; return (
@@ -57,13 +83,17 @@ function TaskTable({ setup, metric }) { sortBy("name")}>Task - sortBy("score")}>Score ({METRIC_LABELS[metric]}) + {/* The metric names the column on its own: "Score (Tokens)" + calls a token count a score, and the parenthetical was + only ever there because "Score" couldn't carry which one. */} + sortBy("score")}>{METRIC_LABELS[metric]} {tasks.map(task => { // Null-safe: an unscored task shows an empty bar and "—". const s = task.scores[metric]; + const barPct = metricBarFraction(metric, s, taskBest) * 100; return ( @@ -75,9 +105,9 @@ function TaskTable({ setup, metric }) {
-
+
- {s == null ? "—" : `${s}%`} + {formatMetric(metric, s)}
@@ -131,17 +161,33 @@ export function Detail() { const model = models[setup.model]; const harness = harnesses[setup.harness]; - const score = setupScore(setup, metric) ?? 0; + const score = setupScore(setup, metric); // Null-safe summary stats: drop tasks with no score for this metric, and // guard the all-empty case so a sparse setup renders "—" instead of NaN / // -Infinity. Mirrors setupScore()'s null handling; `vals.length` is the // number of *scored* tasks, which is what "Average over N tasks" should mean. const vals = setup.tasks.map(t => t.scores[metric]).filter(v => v != null); - const best = vals.length ? Math.max(...vals) : null; + // "Best" follows the metric's direction: the fastest task, not the slowest. + const best = vals.length ? (isLowerBetter(metric) ? Math.min(...vals) : Math.max(...vals)) : null; const avg = vals.length ? vals.reduce((a, b) => a + b, 0) / vals.length : null; const med = vals.length ? median(vals) : null; - const pct = v => (v == null ? "—" : `${v.toFixed(1)}%`); + const pct = v => formatMetric(metric, v); + + // The fifth card reports an efficiency axis the toggle is NOT showing, so it + // adds a number instead of repeating one. This card used to be a hardcoded + // "Avg Speed", which was independent of the metric back when latency wasn't + // selectable; now that it is, selecting Latency makes "Average" the mean + // latency and the two cards print the same figure side by side. Taking the + // first efficiency metric other than the selected one gives Input Tokens + // under Latency and Latency everywhere else, without naming either key here. + const companion = METRICS.find(m => !metricMeta(m).percentage && m !== metric); + const companionVals = companion + ? setup.tasks.map(t => t.scores[companion]).filter(v => v != null) + : []; + const companionAvg = companionVals.length + ? companionVals.reduce((a, b) => a + b, 0) / companionVals.length + : null; return (
@@ -153,9 +199,20 @@ export function Detail() {
-
-
- {score.toFixed(1)}% + {/* At eight metrics the toggle is two buttons wider than it + was, which is enough to clip the longest model name to + "Gamma Co…" — the one string on the page that must not be + abbreviated. Merely making this column shrinkable isn't + enough: flex splits the shortfall in proportion to content + width, so the identity still gives up pixels it can only + pay for by truncating, while the toggle beside it could + have wrapped for free. The lopsided shrink factor says + which item yields — this one, all the way down to the + headline figure (shrink-0, so it never breaks), and only + then does the name start to shorten. */} +
+
+ {formatMetric(metric, score)} {METRIC_LABELS[metric]}
@@ -167,21 +224,35 @@ export function Detail() { - - + {/* Only on the quality metrics. What a catastrophic + violation zeroes is the Outcome score — the seconds and + tokens the run consumed are untouched and still valid, so + in a row of cards that otherwise all describe the selected + metric, this one would read as qualifying a figure it has + no bearing on. Same rule as the leaderboard's ⚠ badge. */} + {metricMeta(metric).percentage && ( + + )} + {companion ? ( + + ) : null}
{/* Task breakdown */} @@ -195,9 +266,9 @@ export function Detail() { - Score Trend Over Time + {METRIC_LABELS[metric]} Trend Over Time -

This setup's success rate across historical run iterations.

+

This setup's {METRIC_LABELS[metric].toLowerCase()} across historical run iterations.

diff --git a/site/src/pages/Detail.test.jsx b/site/src/pages/Detail.test.jsx index c18fe861..b0b47985 100644 --- a/site/src/pages/Detail.test.jsx +++ b/site/src/pages/Detail.test.jsx @@ -1,5 +1,5 @@ import { describe, it, expect, beforeEach, vi } from "vitest"; -import { render, screen, fireEvent, within } from "@testing-library/react"; +import { render, screen, fireEvent, within, cleanup } from "@testing-library/react"; import { MemoryRouter, Routes, Route } from "react-router-dom"; // Stub the chart (jsdom has no canvas); the context is mocked per-test below. @@ -26,10 +26,15 @@ function makeBenchmark(overrides = {}) { { id: SETUP_ID, order: 0, model: "alpha-pro", harness: "gemini-cli", augmentation: [], color: "#3b82f6", + // The efficiency figures are chosen so their means are round + // (50.0s, 20.0k input) and their best is the SMALLEST, which is + // the opposite end from the percentage metrics above. No + // cachedTokens: this harness reports no cache reads, so that + // axis stays null the way a real gemini-cli row does. tasks: [ - { folder: "a", name: "Apple", scores: { composite: 60, pass1: 60, pass5: 65, passMax: 70 } }, - { folder: "b", name: "Banana", scores: { composite: 90, pass1: 90, pass5: 95, passMax: 100 } }, - { folder: "c", name: "Cherry", scores: { composite: 80, pass1: 80, pass5: 85, passMax: 90 } } + { folder: "a", name: "Apple", scores: { composite: 60, pass1: 60, pass5: 65, passMax: 70, latency: 40, inputTokens: 10000, outputTokens: 400 } }, + { folder: "b", name: "Banana", scores: { composite: 90, pass1: 90, pass5: 95, passMax: 100, latency: 50, inputTokens: 20000, outputTokens: 500 } }, + { folder: "c", name: "Cherry", scores: { composite: 80, pass1: 80, pass5: 85, passMax: 90, latency: 60, inputTokens: 30000, outputTokens: 600 } } ], history: [ { t: "2026-01-15T00:00:00Z", scores: { composite: 70, pass1: 70, pass5: 75, passMax: 80 } }, @@ -155,6 +160,56 @@ describe("Detail", () => { expect(within(card("Catastrophic")).getByText("outcomes zeroed")).toBeInTheDocument(); }); + it("drops the catastrophic card on the efficiency metrics", () => { + // A catastrophic violation zeroes the OUTCOME score. The seconds and + // tokens the run consumed are untouched, so beside a latency or token + // headline the card qualifies a figure it has no bearing on. + benchmark.setups[0].catastrophicCount = 3; + for (const m of ["latency", "inputTokens", "outputTokens"]) { + renderAt(`/setup/${SETUP_ID}?metric=${m}`); + expect(screen.queryByText("Catastrophic")).not.toBeInTheDocument(); + cleanup(); + } + // ...and is still there on the quality side, where it explains the score. + renderAt(`/setup/${SETUP_ID}?metric=composite`); + expect(screen.getByText("Catastrophic")).toBeInTheDocument(); + }); + + it("reports the efficiency axis the toggle is not showing", () => { + const card = label => screen.getByText(label).closest("div"); + + // Under a quality metric the spare card is latency, as it always was. + renderAt(`/setup/${SETUP_ID}`); + expect(within(card("Avg Latency")).getByText("50.0s")).toBeInTheDocument(); + + // Under Latency it has to switch, or it prints the same figure as + // "Average" in the card next to it. + cleanup(); + renderAt(`/setup/${SETUP_ID}?metric=latency`); + expect(within(card("Average")).getByText("50.0s")).toBeInTheDocument(); + expect(screen.queryByText("Avg Latency")).not.toBeInTheDocument(); + // Falls to the next efficiency axis in METRICS order, which is now the + // input-token axis rather than a combined token count. + expect(within(card("Avg Input Tokens")).getByText("20.0k")).toBeInTheDocument(); + }); + + it("orients the stat cards by the metric's direction", () => { + // Best is the FASTEST task under latency — the minimum, where every + // percentage metric takes the maximum. + renderAt(`/setup/${SETUP_ID}?metric=latency`); + const card = label => screen.getByText(label).closest("div"); + expect(within(card("Best Task")).getByText("40.0s")).toBeInTheDocument(); + expect(within(card("Median")).getByText("50.0s")).toBeInTheDocument(); + }); + + it("heads the task column with the metric rather than calling it a score", () => { + // A token count is telemetry, not a score; the old "Score (Tokens)" + // header said otherwise. + renderAt(`/setup/${SETUP_ID}?metric=inputTokens`); + expect(screen.getByRole("columnheader", { name: /Tokens/ })).toBeInTheDocument(); + expect(screen.queryByRole("columnheader", { name: /Score/ })).not.toBeInTheDocument(); + }); + it("honors the ?metric= query param", () => { renderAt(`/setup/${SETUP_ID}?metric=pass5`); expect(screen.getByRole("button", { name: "Pass@5" })).toHaveAttribute("aria-pressed", "true"); @@ -181,10 +236,26 @@ describe("Detail", () => { it("toggles score sort direction on repeated header clicks", () => { renderAt(`/setup/${SETUP_ID}`); - fireEvent.click(screen.getByRole("columnheader", { name: /Score/ })); + fireEvent.click(screen.getByRole("columnheader", { name: /Outcome/ })); expect(taskOrder()).toEqual(["Apple", "Cherry", "Banana"]); // now ascending }); + it("points the sort arrow the way the values actually run", () => { + // Both default to best-first, but "best" is the largest percentage and + // the smallest token count, so the same sort state has to draw opposite + // arrows — otherwise a column ascending 10.0k → 30.0k sits under a ▼. + const header = name => screen.getByRole("columnheader", { name }); + + renderAt(`/setup/${SETUP_ID}`); + expect(header(/Outcome/)).toHaveTextContent("▼"); // 90 → 60, descending + + cleanup(); + renderAt(`/setup/${SETUP_ID}?metric=inputTokens`); + expect(header(/Tokens/)).toHaveTextContent("▲"); // 10.0k → 30.0k, ascending + fireEvent.click(header(/Tokens/)); + expect(header(/Tokens/)).toHaveTextContent("▼"); + }); + it("shows a NotFound state for an unknown setup id", () => { renderAt("/setup/does-not-exist"); expect(screen.getByText(/No setup found/i)).toBeInTheDocument(); diff --git a/site/src/pages/Leaderboard.jsx b/site/src/pages/Leaderboard.jsx index 72516b4d..c41da230 100644 --- a/site/src/pages/Leaderboard.jsx +++ b/site/src/pages/Leaderboard.jsx @@ -5,8 +5,8 @@ import { useMemo, useState } from "react"; import { useBenchmark } from "../context/BenchmarkContext.jsx"; import { buildFilterGroups, getFilteredSetups, emptyFilterState } from "../lib/filters.js"; -import { setupScore } from "../lib/accessors.js"; -import { availableMetrics, metricDescription } from "../lib/vocab.js"; +import { setupScore, compareByName } from "../lib/accessors.js"; +import { METRIC_LABELS, availableMetrics, metricDescription, isLowerBetter, bestValue } from "../lib/vocab.js"; import { FilterBar } from "../components/FilterBar.jsx"; import { LeaderboardRow } from "../components/LeaderboardRow.jsx"; import { MetricToggle } from "../components/MetricToggle.jsx"; @@ -17,6 +17,11 @@ export function Leaderboard() { const { models, harnesses, setups, loading, error } = useBenchmark(); const [metric, setMetric] = useState("composite"); const [filterState, setFilterState] = useState(emptyFilterState); + // Rank-first is the default, so the page still opens as a leaderboard. The + // key deliberately SURVIVES a metric change: the reason to sort by name is + // to hold every row still while you click across the metric tabs and read + // one setup down the columns, which re-ranking on each tab makes impossible. + const [sort, setSort] = useState({ key: "metric", dir: "desc" }); const groups = useMemo(() => buildFilterGroups(models, harnesses, setups), [models, harnesses, setups]); const available = useMemo(() => availableMetrics(setups), [setups]); @@ -26,10 +31,33 @@ export function Leaderboard() { [setups, groups, filterState] ); - // Sort the filtered setups by aggregated score under the selected metric. - const sorted = useMemo( - () => [...filtered].sort((a, b) => (setupScore(b, metric) ?? 0) - (setupScore(a, metric) ?? 0)), - [filtered, metric] + // Sort the filtered setups, either alphabetically or by aggregated score + // under the selected metric. "desc" means BEST first, which for latency and + // the token axes is the SMALLEST value — same convention as the detail + // page's task table, so the two cannot disagree about which way is up. + // A setup with no value for the metric sorts last in either direction rather + // than being treated as a 0, which would make it look like the best latency. + const sorted = useMemo(() => { + const dir = sort.dir === "asc" ? 1 : -1; + const lower = isLowerBetter(metric); + return [...filtered].sort((a, b) => { + if (sort.key === "name") return dir * compareByName(a, b, models, harnesses); + const av = setupScore(a, metric); + const bv = setupScore(b, metric); + if (av == null && bv == null) return 0; + if (av == null) return 1; + if (bv == null) return -1; + return dir * (lower ? bv - av : av - bv); + }); + }, [filtered, metric, sort, models, harnesses]); + + // Best value on screen, so an absolute metric's bars have a scale. Computed + // for every metric; a percentage metric ignores it downstream in + // metricBarFraction(). See bestValue() for why non-positive readings are + // excluded rather than min'd over. + const metricBest = useMemo( + () => bestValue(metric, sorted.map(s => setupScore(s, metric))), + [sorted, metric] ); function toggleFilter(groupKey, value) { @@ -45,6 +73,45 @@ export function Leaderboard() { setFilterState(emptyFilterState()); } + // Clicking the header you are already sorted by flips direction; switching + // headers picks that column's natural default — A→Z for names, best-first + // for the metric. + function sortBy(key) { + setSort(prev => prev.key === key + ? { key, dir: prev.dir === "asc" ? "desc" : "asc" } + : { key, dir: key === "name" ? "asc" : "desc" }); + } + + // The glyph describes the column's ORDER, never the raw digits: on the + // metric ▼ is best-first and ▲ is worst-first, on every metric alike. + // + // Pointing it at the digits instead made it flip with no click, because + // best-first is DESCENDING numbers on Outcome but ASCENDING numbers on + // latency and the token axes. Moving between the two families inverted the + // arrow on its own, which reads as the sort mode changing by itself when + // all you did was change tabs. Rank is the stable thing here, so the arrow + // tracks rank; the cost is a ▼ above a latency column that counts upward, + // which the label spells out. + // + // These headers are buttons in a CSS grid, not real columnheaders, so + // aria-sort would be invalid — the direction rides along as + // visually-hidden text in the button's accessible name instead. + const Arrow = ({ k }) => { + if (sort.key !== k) return ; + const down = sort.dir === "desc"; + const directionLabel = k === "name" + ? (down ? "Z to A" : "A to Z") + : (down ? "best first" : "worst first"); + return ( + <> + + , sorted {directionLabel} + + ); + }; + + const headerBtn = "flex items-center gap-1 hover:text-slate-700 dark:hover:text-slate-200 transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-indigo-500 rounded"; + return (
@@ -74,18 +141,31 @@ export function Leaderboard() { {/* Controls & column headers */}
- MODEL + {/* Both halves of the pairing sort the same way — the + comparator runs model then harness — so either one is + a live target rather than only the leading word. */} + - HARNESS & config +
- SCORE -
+ {/* "METRIC", not "SCORE": the toggle below can select + latency or a token axis, none of which is a score. Naming the + selected metric here instead would just echo the + highlighted button an inch beneath it. */} + +
@@ -104,7 +184,7 @@ export function Leaderboard() { : error ? : sorted.length === 0 ? : sorted.map(setup => ( - + ))}
@@ -117,9 +197,9 @@ export function Leaderboard() { - Accuracy Performance Trend Over Time + {METRIC_LABELS[metric]} Trend Over Time -

Comparing agent configuration success rates across historical run iterations.

+

Comparing agent configuration {METRIC_LABELS[metric].toLowerCase()} across historical run iterations.

)} diff --git a/site/src/pages/Leaderboard.test.jsx b/site/src/pages/Leaderboard.test.jsx index f48f7a60..cd3deff2 100644 --- a/site/src/pages/Leaderboard.test.jsx +++ b/site/src/pages/Leaderboard.test.jsx @@ -15,17 +15,20 @@ const FIXTURE = { "openclaw": { name: "OpenClaw", type: "cli", accent: "#f43f5e", logo: "claw" } }, setups: [ + // Deliberately ranked against the alphabet: Gamma Coder leads on + // composite while Alpha Pro leads on latency, so a sort assertion can + // tell rank order, reverse-rank order and name order apart. { id: "alpha-pro-gemini-cli", order: 0, model: "alpha-pro", harness: "gemini-cli", augmentation: [], color: "#3b82f6", - tasks: [{ folder: "a", name: "A", scores: { pass1: 90, pass5: 95, passMax: 100 } }], - history: [{ t: "2026-01-15T00:00:00Z", scores: { pass1: 90, pass5: 95, passMax: 100 } }] + tasks: [{ folder: "a", name: "A", scores: { pass1: 90, pass5: 95, passMax: 100, composite: 70, latency: 20 } }], + history: [{ t: "2026-01-15T00:00:00Z", scores: { pass1: 90, pass5: 95, passMax: 100, composite: 70, latency: 20 } }] }, { id: "gamma-coder-openclaw-mcp-skills", order: 1, model: "gamma-coder", harness: "openclaw", augmentation: ["mcp", "skills"], color: "#ec4899", - tasks: [{ folder: "a", name: "A", scores: { pass1: 70, pass5: 75, passMax: 80 } }], - history: [{ t: "2026-01-15T00:00:00Z", scores: { pass1: 70, pass5: 75, passMax: 80 } }] + tasks: [{ folder: "a", name: "A", scores: { pass1: 70, pass5: 75, passMax: 80, composite: 90, latency: 50, cachedTokens: 5000 } }], + history: [{ t: "2026-01-15T00:00:00Z", scores: { pass1: 70, pass5: 75, passMax: 80, composite: 90, latency: 50, cachedTokens: 5000 } }] } ], loading: false, @@ -67,4 +70,114 @@ describe("Leaderboard", () => { fireEvent.click(pass5); expect(pass5).toHaveAttribute("aria-pressed", "true"); }); + + it("names the selected metric in the trend heading and caption", () => { + // The heading used to be hardcoded "Accuracy Performance Trend Over + // Time", which reads as a falsehood under an efficiency metric where + // the series is seconds or tokens rather than a success rate. + renderPage(); + expect(screen.getByRole("heading", { name: /Outcome Trend Over Time/i })).toBeInTheDocument(); + + fireEvent.click(screen.getByRole("button", { name: "Pass@5" })); + expect(screen.getByRole("heading", { name: /Pass@5 Trend Over Time/i })).toBeInTheDocument(); + expect(screen.queryByText(/success rates/i)).not.toBeInTheDocument(); + }); +}); + +// Row order, read off the links in DOM order. +const order = () => screen.getAllByRole("link").map(a => a.getAttribute("aria-label")); +const modelHeader = () => screen.getByRole("button", { name: /^MODEL/ }); +const metricHeader = () => screen.getByRole("button", { name: /^METRIC/ }); + +describe("Leaderboard sorting", () => { + it("still opens ranked best-first, so it reads as a leaderboard", () => { + renderPage(); + // Gamma leads on composite despite sorting second alphabetically. + expect(order()[0]).toMatch(/Gamma Coder/); + }); + + it("sorts alphabetically when the identity header is clicked", () => { + renderPage(); + fireEvent.click(modelHeader()); + expect(order()[0]).toMatch(/Alpha Pro/); + }); + + it("holds that order across metric tabs, which is the point of having it", () => { + // The reason to sort by name is to read one setup down the columns. + // Re-ranking on every tab makes that impossible, so the sort key has to + // survive a metric change. Composite and latency rank oppositely here, + // so a regression to auto-rank would visibly reorder. + renderPage(); + fireEvent.click(modelHeader()); + for (const m of ["Latency", "Pass@5"]) { + fireEvent.click(screen.getByRole("button", { name: m })); + expect(order()[0]).toMatch(/Alpha Pro/); + } + }); + + it("re-ranks per metric while sorted by rank, the pre-existing behavior", () => { + renderPage(); + expect(order()[0]).toMatch(/Gamma Coder/); + // Best latency is the SMALLEST, so Alpha (20s) leads Gamma (50s). + fireEvent.click(screen.getByRole("button", { name: "Latency" })); + expect(order()[0]).toMatch(/Alpha Pro/); + }); + + it("flips direction when the active header is clicked again", () => { + renderPage(); + fireEvent.click(modelHeader()); + expect(order()[0]).toMatch(/Alpha Pro/); + fireEvent.click(modelHeader()); + expect(order()[0]).toMatch(/Gamma Coder/); + + fireEvent.click(metricHeader()); + expect(order()[0]).toMatch(/Gamma Coder/); + fireEvent.click(metricHeader()); + expect(order()[0]).toMatch(/Alpha Pro/); + }); + + it("never changes sort direction on its own when the metric changes", () => { + // Regression: the arrow used to describe the raw digits, and best-first + // is DESCENDING numbers on composite but ASCENDING numbers on latency. + // Crossing between the two families inverted it with no click, which + // reads as the sort mode switching by itself. + renderPage(); + expect(metricHeader()).toHaveAccessibleName(/best first/); + // Latency/Cached are lower-is-better and Pass@1/Outcome higher — the + // crossing that used to invert the glyph. + for (const m of ["Latency", "Pass@1", "Cached Tokens", "Outcome"]) { + fireEvent.click(screen.getByRole("button", { name: m })); + expect(metricHeader()).toHaveAccessibleName(/best first/); + } + // Only an explicit click moves it, and then it stays moved. + fireEvent.click(metricHeader()); + expect(metricHeader()).toHaveAccessibleName(/worst first/); + fireEvent.click(screen.getByRole("button", { name: "Latency" })); + expect(metricHeader()).toHaveAccessibleName(/worst first/); + }); + + it("keeps the winner on top across every metric, quality or efficiency", () => { + // The arrow no longer tracks the digits, but best-first is still what + // the default MEANS: highest composite, lowest latency. + renderPage(); + expect(order()[0]).toMatch(/Gamma Coder/); // composite 90 > 70 + fireEvent.click(screen.getByRole("button", { name: "Latency" })); + expect(order()[0]).toMatch(/Alpha Pro/); // latency 20s < 50s + }); + + it("gives an identical alphabetical order on every metric tab", () => { + // The name comparator reads model/harness/augmentation only, so the + // ordering cannot vary by metric — including on tabs where a setup has + // no reading at all and the rank sort would shuffle it to the bottom. + renderPage(); + fireEvent.click(modelHeader()); + const baseline = order(); + expect(baseline[0]).toMatch(/Alpha Pro/); + for (const m of ["Latency", "Input Tokens", "Output Tokens", "Cached Tokens", "Pass@1", "Outcome"]) { + const pill = screen.queryByRole("button", { name: m }); + if (!pill || pill.disabled) continue; + fireEvent.click(pill); + expect(order()).toEqual(baseline); + } + }); });