Skip to content

feat(site): surface latency and tokens as leaderboard metrics - #243

Open
jessie1111101 wants to merge 1 commit into
feat/site-dark-modefrom
feat/site-efficiency-metrics
Open

feat(site): surface latency and tokens as leaderboard metrics#243
jessie1111101 wants to merge 1 commit into
feat/site-dark-modefrom
feat/site-efficiency-metrics

Conversation

@jessie1111101

Copy link
Copy Markdown
Collaborator

Summary

Efficiency data was already in Firestore and the dashboard could never show it. Every ResultRow carries latencySec and the token buckets, but derive() didn't project them into setups — and the browser only reads setups. This adds them as two metrics alongside the quality scores.

No producer or ingest changes needed. The data has been landing in the results collection all along; the missing link was one function plus the UI's percentage assumptions.

What's here

  • derive() (and its seeder mirror) average latency and total tokens per task and per history point. Efficiency is treated as telemetry, not a score: it's averaged over all rows rather than the scored subset, and it survives a cell where nothing scored — an unscored iteration still burned wall-clock and tokens.
  • Metrics now carry presentation rules (unit, lowerIsBetter, percentage). Quality metrics keep their existing behaviour verbatim; latency and tokens format by unit (42.7s, 38.4k), rank ascending, and scale their bar against the largest value on screen — inverted, so the fastest setup still gets the fullest bar and "longer is better" stays true everywhere.
  • Direction is respected throughout: leaderboard sort, the detail page's Best Task card (which would otherwise show the slowest task), and the trend chart's y-axis, which previously clamped to [0,100] and would have flattened every latency series onto the top gridline.
  • Tokens prefer the producer's own totalTokens when present and otherwise sum the captured buckets, staying null when the harness recorded no usage — so "not measured" never renders as zero.

Notes

Test plan

  • vitest107 passed (12 files), including new coverage for the formatter, direction, and bar-scaling
  • vite build --mode staging clean
  • Seeded the emulator and confirmed latency and tokens now appear on all derived setup docs (132 occurrences each)

The rows already carried latencySec and the token buckets; derive() dropped
them, so the dashboard could never show efficiency. Project them into Scores and
teach the UI that not every metric is a percentage.

- derive (and its seeder mirror) average latency and total tokens per task and
  per history point. Efficiency is telemetry, not a score: it is averaged over
  ALL rows and survives a cell where nothing scored, since an unscored iteration
  still consumed time and tokens.
- A metric now carries presentation rules (unit, direction, percentage?). The
  quality metrics keep the old behaviour; latency and tokens format by unit,
  rank ascending, and scale their bar against the largest value on screen,
  inverted so the fastest setup still gets the fullest bar.
- Sorting, the detail page's Best-Task card, and the chart's y-axis all follow
  the metric's direction rather than assuming higher-is-better in [0,100].

Tokens prefer the producer's own total when present and otherwise sum the
captured buckets, staying null when the harness recorded no usage so 'not
measured' never reads as zero.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant