feat: canonical token buckets across harnesses and the result row - #212
Open
eugeneng04 wants to merge 3 commits into
Open
feat: canonical token buckets across harnesses and the result row#212eugeneng04 wants to merge 3 commits into
eugeneng04 wants to merge 3 commits into
Conversation
Proposes one canonical six-bucket token schema (input, cached, cache_write, reasoning, output, total; None = unreported) that every harness maps onto, with per-provider normalization rules anchored on 'input excludes cached tokens'. Covers the row-schema widening (cached/reasoning on ResultRow + normalizer aliases), moving the shared shape out of the antigravity harness, migration for historical rows, and open questions. The antigravity harness already emits this shape; this design promotes it to the shared contract.
Implements the unified token accounting design (docs/designs/token-accounting.md): one canonical six-bucket token schema — input (non-cached), cached, cache_write, reasoning, output (excludes reasoning), total — with None for unreported buckets, never a fabricated 0. - agents/result.py: shared TOKEN_BUCKETS + empty_tokens(). - api harness: extract_tokens canonicalizes each provider shape — Anthropic input_tokens passes through (already non-cached) with cache read/write buckets; Gemini subtracts cached_content_token_count (a subset of prompt_token_count) and adds tool_use_prompt tokens; OpenAI subtracts prompt_tokens_details.cached_tokens and splits completion_tokens_details.reasoning_tokens out of output. Total prefers the provider total, else the bucket sum. - gemini CLI harness: the terminal result.stats block maps to the canonical dict (input = full input − cached; reasoning derived from the total gap). - results/normalize.py + ResultRow: rows carry cachedTokens/reasoningTokens (additive nullable fields — no SCHEMA_VERSION bump; legacy provider aliases keep historical results.json readable); dashboard schema.d.ts and the ingest validator accept the new fields absent-tolerantly. - openclaw still passes provider-native usage through (aliases flatten it; cached/reasoning None until canonicalized); antigravity emits the shape via its own decoder (gke-labs#211).
eugeneng04
changed the title
Review follow-up: the leaderboard cost formula (input*p_in + cached*p_cached + cache_write*p_write + output*p_out) needs cache-creation counts, which bill at a premium — dropping them undercounts cache-heavy runs. normalize_tokens now returns all six buckets (with legacy total_tokens/total_token_count and cache_creation_input_tokens aliases for historical records), ResultRow/schema.d.ts gain cacheWriteTokens/totalTokens as additive nullable fields, and the ingest validator accepts them absent-tolerantly. Note: total semantics vary for pre-canonical records (some eras exclude cached/reasoning); documented on the row.
eugeneng04
added a commit
to eugeneng04/devops-bench
that referenced
this pull request
Jul 20, 2026
Map Anthropic usage onto the canonical six-bucket shape (input, cached, cache_write, reasoning, output, total; None = unreported): input_tokens passes through as the non-cached prompt, cache reads and cache writes become separate buckets (writes bill at a premium, so folding them into cached — the previous shape — mis-prices cache-heavy runs), reasoning stays None (Anthropic bills thinking inside output_tokens), and total is the full footprint. The per-turn accumulator fallback produces the same shape, empty/no-usage streams yield all-None (never a fabricated 0), and errored paths carry the canonical shape too. Bucket tuple is harness-local until the unified token schema (gke-labs#212) lands. Also trims narration comments in the parser and agent.
eugeneng04
added a commit
to eugeneng04/devops-bench
that referenced
this pull request
Jul 22, 2026
Map Anthropic usage onto the canonical six-bucket shape (input, cached, cache_write, reasoning, output, total; None = unreported): input_tokens passes through as the non-cached prompt, cache reads and cache writes become separate buckets (writes bill at a premium, so folding them into cached — the previous shape — mis-prices cache-heavy runs), reasoning stays None (Anthropic bills thinking inside output_tokens), and total is the full footprint. The per-turn accumulator fallback produces the same shape, empty/no-usage streams yield all-None (never a fabricated 0), and errored paths carry the canonical shape too. Bucket tuple is harness-local until the unified token schema (gke-labs#212) lands. Also trims narration comments in the parser and agent.
jessie1111101
added a commit
that referenced
this pull request
Jul 22, 2026
Adds turns (agent trajectory steps = tool calls + text turns) as a per-run efficiency field, unblocking the Phase 2 efficiency axes. - results/row.py: ResultRow.turns (int | None); normalize.py derives it from the record's trajectory length (None when no trajectory captured, 0 for a run that took no steps). - site: schema.d.ts + load.mjs (optional validation) + PROTOCOL.md document it. - tests: normalize turns counting + loader validation. Independent of the token-buckets work (#212) — turns comes from the trajectory, not token usage — but touches the same row/normalize/schema/load files, so expect a trivial rebase on whichever of {this, #212, #195, #206} lands second. pytest results green; site vitest 97/97.
jessie1111101
approved these changes
Jul 22, 2026
jessie1111101
left a comment
jessie1111101
left a comment
Collaborator
There was a problem hiding this comment.
lgtm, but we may want to port this PR to kubernetes-sigs rather than merge here
Collaborator
Author
Migration note: partial port to kubernetes-sigsThe portable slice of this PR is up as kubernetes-sigs/devops-bench#38 — the parts that touch modules already migrated upstream:
Still to port once their modules migrate (they target code not yet on kubernetes-sigs):
When those modules land upstream, port the remaining hunks so the token schema is complete there. |
jessie1111101
added a commit
that referenced
this pull request
Jul 28, 2026
Adds turns (agent trajectory steps = tool calls + text turns) as a per-run efficiency field, unblocking the Phase 2 efficiency axes. - results/row.py: ResultRow.turns (int | None); normalize.py derives it from the record's trajectory length (None when no trajectory captured, 0 for a run that took no steps). - site: schema.d.ts + load.mjs (optional validation) + PROTOCOL.md document it. - tests: normalize turns counting + loader validation. Independent of the token-buckets work (#212) — turns comes from the trajectory, not token usage — but touches the same row/normalize/schema/load files, so expect a trivial rebase on whichever of {this, #212, #195, #206} lands second. pytest results green; site vitest 97/97.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements one canonical token schema across the agent harnesses and carries the new buckets through to the result row. (First commit is the design doc; second is the implementation.)
{"input", "cached", "cache_write", "reasoning", "output", "total"} # None = unreported, never a fabricated 0Why
inputmeans different things per provider today: Anthropic reports the uncached prompt, while OpenAI/Gemini report the full prompt including cache hits — so raw cross-harness token/cost comparisons are systematically skewed, and with prompt caching on, cached tokens dominate a multi-turn run's prompt.results/normalize.pyflattened everything toinput/output, droppingcached/reasoning at the row layer.The anchor rule:
inputexcludes cached tokens (Anthropic's convention, and the one usage leaderboards report).What's implemented
agents/result.py:TOKEN_BUCKETS+empty_tokens().apiharness —extract_tokenscanonicalizes each provider shape: Anthropic passes through (already non-cached) with cache read/write buckets; Gemini subtractscached_content_token_count(a documented subset ofprompt_token_count) and adds tool-result input tokens; OpenAI subtractsprompt_tokens_details.cached_tokensand splitsreasoning_tokensout ofcompletion_tokens. Total prefers the provider total, else the bucket sum.geminiCLI harness — the terminalresult.statsblock maps to the canonical dict (input = full input − cached, empirically a subset;reasoningderived from the total gap).ResultRowgainscachedTokens/reasoningTokens(additive nullable fields, noSCHEMA_VERSIONbump; historical rows stay valid),normalize_tokensreads the canonical keys first with legacy provider aliases kept for oldresults.jsonrecords, and the dashboardschema.d.ts+ ingest validator accept the new fields absent-tolerantly.Not in this PR
openclawstill passes provider-native usage through (the aliases flatten it;cached/reasoningstayNoneuntil it is canonicalized).antigravityemits the canonical shape via its own decoder (feat(antigravity): read token usage incl. cached from conversation DB #211); folding its local helper into the shared one is a one-line follow-up once both land.Verification
site/ingest/load.mjschange mirrors the existingintOrNullpattern (absent-tolerant for historical rows); JS tests not run locally (no node) — CI covers them.Design rationale and provider-convention verification:
docs/designs/token-accounting.md(included). Related: #211.