feat(site): surface latency and tokens as leaderboard metrics

site/.firebaserc

@@ -1,5 +1,5 @@
 {
   "projects": {
-    "default": "devops-bench-demo"
+    "default": "devops-bench-shared"
   }
 }

site/README.md

@@ -293,15 +293,35 @@ Needs **Java** for the Firestore emulator
 
 ```bash
 # 1. start the emulator (Firestore :8080, UI :4000)
-npx -y firebase-tools emulators:start --only firestore --project devops-bench-demo
+npx -y firebase-tools emulators:start --only firestore --project devops-bench-shared
 
 # 2. in another shell: seed the emulator (writes to the leaderboard-test DB)
-cd seed && FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-demo npm run seed && cd ..
+cd seed && FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-shared npm run seed && cd ..
 
 # 3. run the app — firebase.js auto-connects to the emulator on localhost
 npm run dev          # http://localhost:5173
 ```
 
+> **The project id must match `VITE_FIREBASE_PROJECT_ID` in `.env`** (that is
+> `devops-bench-shared`, the same project named at the top of this section). The
+> emulator keeps each project id in a **separate namespace**, so seeding one id
+> while the app reads another leaves the dashboard **silently empty** — no error,
+> just no rows. Four places name the project and all four have to agree:
+>
+> - the `--project` flag in step 1
+> - `GCLOUD_PROJECT` in step 2
+> - `VITE_FIREBASE_PROJECT_ID` in `.env`, which is what the browser reads
+> - `"default"` in `.firebaserc`, which is where the emulator falls back if you
+>   drop `--project` — get this one wrong and the emulator UI on :4000 shows no
+>   docs even though the seed reported success
+>
+> If that happens, query the emulator directly to see which namespace the data
+> actually landed in:
+>
+> ```bash
+> curl -s "http://127.0.0.1:8080/v1/projects/devops-bench-shared/databases/leaderboard-test/documents/setups" | head
+> ```
+
 ### B) Staging (real cloud DB, fabricated data)
 
 > ✅ `leaderboard-test` is **already created and seeded** — just run the dev server:
@@ -344,6 +364,7 @@ npm test             # Vitest — fast, DB-free unit + component tests
 | `src/pages/Detail.test.jsx` | stat-card math (incl. null-safe / empty), task-table sorting, `?metric=` param, not-found/loading/error |
 | `src/hooks/useBenchmarkData.test.js` | load-once lifecycle, error capture, terminate-on-PROD |
 | `src/components/TrendChart.test.jsx` | sr-only a11y table: date-union columns, `—` for missing runs and null values |
+| `src/components/MetricToggle.test.jsx` | a button per vocab metric, the quality/efficiency group split, `available` disabling |
 
 ---
 

site/ingest/PROTOCOL.md

@@ -62,9 +62,17 @@ always `0`; the schema is already shaped for multi-iteration runs (§4).
 | `catastrophic` | boolean (optional) | `true` \| `false` | Whether a catastrophic tripwire fired (`cat_v = 0`), zeroing the outcome. Omitted by pre-v1 rows. |
 | `scoringVersion` | string (optional) | e.g. `"v1"` | Scoring-framework version that produced `outcomeScore`. Omitted by pre-v1 rows. |
 | `toolScore` | number \| null | `[0, 1]` or null | Tool-invocation score; `null` when unscored. |
-| `latencySec` | number | `>= 0` | Agent wall-clock latency, seconds. |
+| `latencySec` | number | `>= 0` | Agent wall-clock latency, seconds. Non-nullable, so the producer emits **`0` when latency was not measured**; derive reads `0` as that sentinel rather than as an instant run, and the `Latency` metric treats the row as missing data. |
 | `inputTokens` | integer \| null | `>= 0` or null | Prompt tokens consumed; `null` when usage was not captured. |
 | `outputTokens` | integer \| null | `>= 0` or null | Completion tokens produced; `null` when usage was not captured. |
+| `cachedTokens` | integer \| null (optional) | `>= 0` or null | Cache-read input tokens. Omitted by pre-v1 rows. |
+| `reasoningTokens` | integer \| null (optional) | `>= 0` or null | Reasoning/thinking tokens. A **sibling** bucket of `outputTokens`, not a subset, so the `Tokens` metric adds it in. Omitted by pre-v1 rows. |
+| `cacheWriteTokens` | integer \| null (optional) | `>= 0` or null | Cache-creation input tokens. Omitted by pre-v1 rows. |
+| `totalTokens` | integer \| null (optional) | `>= 0` or null | Provider-reported total. **Preferred over the bucket sum** when present, since it may count buckets the row does not break out. Omitted by pre-v1 rows. |
+
+The `Tokens` leaderboard metric is `totalTokens` when reported, else the sum of
+whichever buckets are present, else `null` (no usage captured). A row reporting
+neither a total nor any bucket is "not measured", not zero tokens.
 
 `setupId`, `model`, `harness`, `augmentation`, `runId`, `t` are **denormalized
 onto every row** — they repeat across all of a run's rows (they mirror the run's

site/ingest/README.md

@@ -88,11 +88,11 @@ brew install openjdk
 export PATH="/opt/homebrew/opt/openjdk/bin:$PATH"
 
 # 2. Start the emulator (from site/, leave it running in another terminal):
-cd .. && npx -y firebase-tools emulators:start --only firestore --project devops-bench-demo
+cd .. && npx -y firebase-tools emulators:start --only firestore --project devops-bench-shared
 # Firestore on :8080, emulator UI on :4000
 
 # 3. Ingest against it (from site/ingest/):
-FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-demo \
+FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-shared \
   node ingest.mjs fixtures/
 ```
 
@@ -131,7 +131,7 @@ or `catalog.mjs` presentation? Re-score every setup from the **existing** raw
 rows — no re-upload (emulator from Option A still running, or ADC for real Firestore):
 
 ```bash
-FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-demo \
+FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-shared \
   node derive.mjs
 ```
 

site/ingest/derive.mjs

@@ -12,18 +12,19 @@
 //   tasks   <- distinct row.taskFolder at the LATEST run of each setup
 //   history <- one aggregate point per distinct row.t, time-ordered
 //
-// The SCORING FORMULA is NOT duplicated here — PASS_THRESHOLD and the pass@k
-// estimator are imported from seed/mock-data.mjs so test data and real data are
-// scored by exactly one definition. Change the formula there and re-run derive
-// (see the CLI at the bottom) to re-score everything from the same raw rows.
+// The SCORING FORMULA is NOT duplicated here — PASS_THRESHOLD, the pass@k
+// estimator and the efficiency projection are imported from seed/mock-data.mjs
+// so test data and real data are scored by exactly one definition. Change the
+// formula there and re-run derive (see the CLI at the bottom) to re-score
+// everything from the same raw rows.
 //
 // Presentation (order / color) is not derivable from results — it's curation, so
 // it comes from the optional catalog overrides, falling back to discovery order
 // and a palette. Model/harness display metadata is handled separately by the
 // catalog (see collectMetadata in catalog.mjs); this module only emits setups.
 // =============================================================================
 
-import { PASS_THRESHOLD, passAtK } from "../seed/mock-data.mjs";
+import { PASS_THRESHOLD, efficiencyFor, passAtK } from "../seed/mock-data.mjs";
 import { PALETTE, SETUP_CATALOG } from "./catalog.mjs";
 
 /**
@@ -55,8 +56,16 @@ function round(v, dp) {
 function scoresFor(rows) {
     const scored = rows.filter(r => Number.isFinite(r.outcomeScore));
     const n = scored.length;
+    // Efficiency is telemetry, not a score: it is recorded even for an iteration
+    // that never scored, so it is averaged over ALL rows rather than the scored
+    // subset, and it survives the no-scored-rows early return below.
+    const efficiency = efficiencyFor(rows);
     if (n === 0) {
-        return { pass1: null, pass5: null, passMax: null, composite: null, correctness: null, recoverableSafety: null };
+        return {
+            pass1: null, pass5: null, passMax: null,
+            composite: null, correctness: null, recoverableSafety: null,
+            ...efficiency
+        };
     }
     // pass1 thresholds on CORRECTNESS `c` (falling back to outcomeScore for
     // pre-v1 rows) so the pass rate isn't distorted by the √/gate composite.
@@ -77,7 +86,8 @@ function scoresFor(rows) {
         passMax: null,
         composite: mean("outcomeScore"),
         correctness: mean("correctnessScore"),
-        recoverableSafety: mean("recoverableSafetyScore")
+        recoverableSafety: mean("recoverableSafetyScore"),
+        ...efficiency
     };
 }
 
@@ -95,7 +105,11 @@ function meanScores(scoreList) {
         passMax: avg("passMax"),
         composite: avg("composite"),
         correctness: avg("correctness"),
-        recoverableSafety: avg("recoverableSafety")
+        recoverableSafety: avg("recoverableSafety"),
+        latency: avg("latency"),
+        inputTokens: avg("inputTokens"),
+        outputTokens: avg("outputTokens"),
+        cachedTokens: avg("cachedTokens")
     };
 }
 
@@ -201,7 +215,7 @@ export function derive(rows, opts = {}) {
 // re-score every setup from the existing raw rows WITHOUT re-uploading. The
 // normal path (ingest.mjs) runs derive automatically after each upload.
 //
-//   FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-demo \
+//   FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 GCLOUD_PROJECT=devops-bench-shared \
 //     node derive.mjs
 //   GCLOUD_PROJECT=devops-bench-shared FIRESTORE_DATABASE_ID=leaderboard-test \
 //     node derive.mjs

site/ingest/derive.test.mjs

@@ -81,7 +81,84 @@ describe("derive — data-driven", () => {
             passMax: null,
             composite: null,
             correctness: null,
-            recoverableSafety: null
+            recoverableSafety: null,
+            // Efficiency is telemetry, not a score: an iteration that never
+            // scored still consumed wall-clock, so latency survives while every
+            // score is null. The token axes stay null because the fixture
+            // captured no usage.
+            latency: 1,
+            inputTokens: null,
+            outputTokens: null,
+            cachedTokens: null
         });
     });
+
+    it("treats latencySec 0 as unmeasured, so it can't rank as the fastest", () => {
+        // Regression: latencySec is non-nullable upstream and normalize.py
+        // coerces a missing measurement to 0.0. Averaging that in gave an
+        // unmeasured setup a latency of 0 — first place on a lower-is-better
+        // metric, with a full bar.
+        const base = {
+            setupId: "s", model: "m", harness: "h", augmentation: [],
+            runId: "run_20260101_000000", t: "2026-01-01T00:00:00Z",
+            taskFolder: "task-a", taskName: "Task A", status: "success",
+            toolScore: null, inputTokens: null, outputTokens: null, outcomeScore: 0.9
+        };
+        const unmeasured = derive([{ ...base, iteration: 0, latencySec: 0 }]);
+        expect(unmeasured[0].tasks[0].scores.latency).toBeNull();
+
+        // A 0 alongside real readings drops out of the mean rather than halving it.
+        const mixed = derive([
+            { ...base, iteration: 0, latencySec: 0 },
+            { ...base, iteration: 1, latencySec: 10 }
+        ]);
+        expect(mixed[0].tasks[0].scores.latency).toBe(10);
+    });
+
+    it("projects the token buckets onto three axes rather than one total", () => {
+        // Regression: a single summed figure was ~the input count wearing a
+        // different label. Output was 0.7% of the fleet's summed buckets, so the
+        // most expensive axis was invisible in the number that ranked setups.
+        const base = {
+            setupId: "s", model: "m", harness: "h", augmentation: [],
+            runId: "run_20260101_000000", t: "2026-01-01T00:00:00Z",
+            taskFolder: "task-a", taskName: "Task A", status: "success",
+            toolScore: null, latencySec: 5, outcomeScore: 0.9, iteration: 0
+        };
+        const split = derive([
+            {
+                ...base,
+                inputTokens: 100,
+                cacheWriteTokens: 50,
+                outputTokens: 200,
+                reasoningTokens: 700,
+                cachedTokens: 4000,
+                // A provider total no longer overrides the buckets: it cannot be
+                // attributed to an axis.
+                totalTokens: 950
+            }
+        ]);
+        expect(split[0].tasks[0].scores).toMatchObject({
+            inputTokens: 150,
+            outputTokens: 900,
+            cachedTokens: 4000
+        });
+    });
+
+    it("leaves the cached axis null for a harness that reports no cache reads", () => {
+        // Only some harnesses report cache reads. A blank cell has to stay
+        // distinct from a 0, or a silent harness would rank best on a
+        // lower-is-better axis purely for being less talkative.
+        const setups = derive([
+            {
+                setupId: "s", model: "m", harness: "h", augmentation: [],
+                runId: "run_20260101_000000", t: "2026-01-01T00:00:00Z",
+                taskFolder: "task-a", taskName: "Task A", status: "success",
+                toolScore: null, latencySec: 5, outcomeScore: 0.9, iteration: 0,
+                inputTokens: 100, outputTokens: 200
+            }
+        ]);
+        expect(setups[0].tasks[0].scores.cachedTokens).toBeNull();
+        expect(setups[0].tasks[0].scores.inputTokens).toBe(100);
+    });
 });

site/ingest/firestore.mjs

@@ -35,7 +35,7 @@ export function openDb() {
     const projectId =
         process.env.GCLOUD_PROJECT ||
         process.env.GOOGLE_CLOUD_PROJECT ||
-        (emulator ? "devops-bench-demo" : "devops-bench-shared");
+        "devops-bench-shared";
     const databaseId = process.env.FIRESTORE_DATABASE_ID || "leaderboard-test";
 
     if (databaseId === PROD_DATABASE_ID && process.env.ALLOW_PROD_INGEST !== "true") {

site/ingest/load.mjs

@@ -83,9 +83,17 @@ export function validateRow(row) {
     if (!STATUSES.has(row.status)) errs.push('status: must be "success" or "failed"');
     floatOrNull("outcomeScore", num01);
     floatOrNull("toolScore", num01);
+    // 0 is in contract: the producer's latencySec is non-nullable and coerces an
+    // unmeasured run to 0.0, which derive reads as the "not measured" sentinel (§2).
     float("latencySec", nonNeg);
     intOrNull("inputTokens", nonNeg);
     intOrNull("outputTokens", nonNeg);
+    // Extra usage buckets — OPTIONAL (pre-v1 rows omit them), validated when
+    // present because derive's token total reads them. A negative value here
+    // would drag the mean below zero and collapse every token bar to empty.
+    for (const k of ["cachedTokens", "reasoningTokens", "cacheWriteTokens", "totalTokens"]) {
+        if (k in row) intOrNull(k, nonNeg);
+    }
 
     // Scoring-framework v1 fields — OPTIONAL (pre-v1 rows omit them). Validate the
     // shape only when present so old runs still ingest.

site/ingest/load.test.mjs

@@ -82,6 +82,28 @@ describe("validateRow", () => {
         expect(errs.join()).toMatch(/inputTokens/);
     });
 
+    it("accepts the extra usage buckets, present or absent", () => {
+        // Optional: pre-v1 rows omit them entirely and must still ingest.
+        expect(validateRow(validRow)).toEqual([]);
+        expect(validateRow({
+            ...validRow,
+            cachedTokens: 100, reasoningTokens: 700, cacheWriteTokens: 0, totalTokens: 9520
+        })).toEqual([]);
+        expect(validateRow({
+            ...validRow,
+            cachedTokens: null, reasoningTokens: null, cacheWriteTokens: null, totalTokens: null
+        })).toEqual([]);
+    });
+
+    it("flags a negative or non-integer usage bucket", () => {
+        // Regression: a negative total reached the token mean and dragged
+        // the token mean below zero, collapsing every token bar to empty.
+        expect(validateRow({ ...validRow, totalTokens: -5 }).join()).toMatch(/totalTokens/);
+        expect(validateRow({ ...validRow, reasoningTokens: 1.5 }).join()).toMatch(/reasoningTokens/);
+        expect(validateRow({ ...validRow, cachedTokens: -1 }).join()).toMatch(/cachedTokens/);
+        expect(validateRow({ ...validRow, cacheWriteTokens: "1000" }).join()).toMatch(/cacheWriteTokens/);
+    });
+
     it("rejects a non-object", () => {
         expect(validateRow(null)).toEqual(["not an object"]);
         expect(validateRow([])).toEqual(["not an object"]);