feat(site): surface latency and tokens as leaderboard metrics - #243
Open
jessie1111101 wants to merge 1 commit into
Open
feat(site): surface latency and tokens as leaderboard metrics#243jessie1111101 wants to merge 1 commit into
jessie1111101 wants to merge 1 commit into
Conversation
The rows already carried latencySec and the token buckets; derive() dropped them, so the dashboard could never show efficiency. Project them into Scores and teach the UI that not every metric is a percentage. - derive (and its seeder mirror) average latency and total tokens per task and per history point. Efficiency is telemetry, not a score: it is averaged over ALL rows and survives a cell where nothing scored, since an unscored iteration still consumed time and tokens. - A metric now carries presentation rules (unit, direction, percentage?). The quality metrics keep the old behaviour; latency and tokens format by unit, rank ascending, and scale their bar against the largest value on screen, inverted so the fastest setup still gets the fullest bar. - Sorting, the detail page's Best-Task card, and the chart's y-axis all follow the metric's direction rather than assuming higher-is-better in [0,100]. Tokens prefer the producer's own total when present and otherwise sum the captured buckets, staying null when the harness recorded no usage so 'not measured' never reads as zero.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Efficiency data was already in Firestore and the dashboard could never show it. Every
ResultRowcarrieslatencySecand the token buckets, butderive()didn't project them intosetups— and the browser only readssetups. This adds them as two metrics alongside the quality scores.No producer or ingest changes needed. The data has been landing in the
resultscollection all along; the missing link was one function plus the UI's percentage assumptions.What's here
derive()(and its seeder mirror) average latency and total tokens per task and per history point. Efficiency is treated as telemetry, not a score: it's averaged over all rows rather than the scored subset, and it survives a cell where nothing scored — an unscored iteration still burned wall-clock and tokens.unit,lowerIsBetter,percentage). Quality metrics keep their existing behaviour verbatim; latency and tokens format by unit (42.7s,38.4k), rank ascending, and scale their bar against the largest value on screen — inverted, so the fastest setup still gets the fullest bar and "longer is better" stays true everywhere.[0,100]and would have flattened every latency series onto the top gridline.totalTokenswhen present and otherwise sum the captured buckets, stayingnullwhen the harness recorded no usage — so "not measured" never renders as zero.Notes
mainas those land.nullas0and making it look like the best latency.turnsis deliberately not included — it's still unmerged in feat(results): emit 'turns' efficiency axis on the result row #215. It slots into the same machinery as a third key when that lands.Test plan
vitest— 107 passed (12 files), including new coverage for the formatter, direction, and bar-scalingvite build --mode stagingcleanlatencyandtokensnow appear on all derived setup docs (132 occurrences each)