MAF-20805: feat(deploy): chart token-weighted tokenizer cache reuse

[bongwoobak]

JIRA: MAF-20805
Artifact: Token-weighted cache reuse

Gateway dependency: needs an AIGateway build that exports the new aigateway_tokenizer_cache_encoded_tokens_total counter and the renamed aigateway_tokenizer_cache_lookup_* counters.

What this PR does

It changes panels 77, 78 and 79 in the Tokenizer cache row of the mif-aigateway dashboard to token-weighted metrics.

The problem

Panel 77, "Cache hit ratio by model", plotted the per-lookup ratio hits / (hits + misses). A lookup counts as one hit even when it reuses only one token. Every chat prompt starts with the same special token, so the panel showed close to 100% for every model, even for long prompts that reused only that one token.

The fix

Panel	After this change	On older gateways
77	Token-weighted cache reuse by model: sum by (model) (rate(hit_prefix_tokens_total) and rate(encoded_tokens_total)) / sum by (model) (rate(encoded_tokens_total)). The and keeps the numerator to pods that also export the denominator, so old pods in a rolling update do not push the ratio above 100%.	Empty. It does not fall back to the old ratio.
78	Cache lookups per second by model: sum by (model) (rate(lookup_*_total) or rate(*_total)). The fallback to the old names is applied per pod, before sum by (model), so old and new pods of the same model are both counted.	Still shown, using the old names.
79	Reused tokens over the same per-pod fallback for hits	Still shown, using the old names.

Panels 77 and 78 now have a description that explains what they measure.

Verification

hack/dashboards/verify_aigateway_tokenizer_cache.py on head 2ccf70e (06bcc98 with main merged in; the dashboard and the script are unchanged by the merge), Prometheus 3.15.0. Fake gateways: model A has one new pod and one old pod, model B has an old pod only.

Query	Result	Expected
77 A	{A: 0.02}	new pod only: 20 / 1000
78 A (hits)	{A: 40, B: 5}	10 new + 30 old
78 B (misses)	{A: 20, B: 5}	10 new + 10 old
79 A	{A: 38, B: 10}	(20 + 1500) / 40

With the queries from 4e5771f, where the fallback is applied after sum by (model) and panel 77 has no and, all 4 checks fail: 1.52, 10, 10 and 152.

The JSON was rewritten with json.dumps(indent=2). I first checked that the file before the change was byte-identical to this format, so the diff shows only these panels.

Copies of this dashboard

Copies of this dashboard in other deployments are updated separately and tracked in Jira.

🤖 Generated with Claude Code

[bongwoobak]

Self-review

[Copilot]

Copilot review overview

🟡 Changes recommended

Apply fallbacks per scrape target before aggregation to handle mixed-version rollouts correctly; clarify the panel description.

Review effort: Lite
Findings: 2 · 1

Open (3)

• Apply fallback before aggregation to preserve mixed-version series · New
• Apply denominator fallback per target before model aggregation · New
• Clarify metric description for cached-prefix lookup matches · New

What changed in this PR

Updates the AIGateway Grafana dashboard with token-weighted tokenizer cache metrics and legacy metric fallbacks.

Changes:

• Replaces lookup-based cache ratio with token-weighted reuse.
• Adds renamed metric queries with legacy fallbacks.
• Updates descriptions for panels 77–79.

File	Summary
deploy/​helm/​moai-inference-framework/​files/​dashboards/​mif-aigateway.json	Updates tokenizer cache dashboard panels and PromQL queries.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟢 Approval recommended

No unresolved review issues were identified.

Review effort: Lite
Findings: None

Resolved since last review (3)

• Apply denominator fallback per target before model aggregation
• Apply fallback before aggregation to preserve mixed-version series
• Clarify metric description for cached-prefix lookup matches

[Copilot]

Copilot review overview

🟢 Approval recommended

All reviewed changes have no unresolved blocking issues.

Review effort: Lite
Findings: None

[hhk7734]

The query changes look correct. I ran hack/dashboards/verify_aigateway_tokenizer_cache.py on 06bcc98 with Prometheus 3.15.0: all 4 checks pass. With the queries from 4e5771f, all 4 fail (1.52, 10, 10, 152), as the description says. The changes below are about the PR text and the artifact, not the code.

Note: mif is a public repository, so I rewrote the description in English. I also removed links to internal repositories, internal hostnames, production values and other ticket keys. Please keep them in Jira. I did not change the structure, so the points below still apply to the current text.

1. Start the description with the JIRA and Artifact lines.
Put these two lines first, before the summary. Move the Jira: MAF-20805 line from the bottom into the first one:

JIRA: [MAF-20805](https://moreh.atlassian.net/browse/MAF-20805)
Artifact: [<title>](https://claude.ai/artifact/Uworr7xayd1ir6ViixxfFc)

2. Update the Verification section to match head.
The numbers come from the first manual run, where the new and old exporters were different models (78: 10/0 and 30/10, 79: 1.0 and 100.0). Your own thread says this setup missed the case where one model has both a new and an old pod. Please replace them with the script output on head, and write the head SHA next to them. On 06bcc98 I got:

• 77 A: {A: 0.02}
• 78 A: {A: 40, B: 5}
• 78 B: {A: 20, B: 5}
• 79 A: {A: 38, B: 10}

Also, the panel 77 row in the table shows rate(hit_prefix_tokens_total) / rate(encoded_tokens_total). It does not show the and encoded_tokens_total limit added in 6f51c1c. Please add it. For panels 78 and 79, say that the fallback is applied per pod, before sum by (model).

3. Update the artifact to match head.
The artifact the description links does not show a head SHA. It shows panel 77 without the and limit, and it does not mention the per-pod fallback fix (f4fdb09, 6f51c1c) or the verification script. Its verification section describes the same earlier run with separate exporters. Please bring it up to 06bcc98 (or the new head), with the final queries and the script result.

[bongwoobak]

@hhk7734 All three points are addressed.

1. JIRA and Artifact lines now open the description, and the Jira: MAF-20805 line at the bottom is gone.
2. Verification now lists the script output on head 2ccf70e (06bcc98 with main merged in; the dashboard and the script did not change): 77 A {A: 0.02}, 78 A {A: 40, B: 5}, 78 B {A: 20, B: 5}, 79 A {A: 38, B: 10}. The same checks against 4e5771f fail with 1.52, 10, 10 and 152. The fix table now shows the and encoded_tokens_total limit on panel 77, and says that panels 78 and 79 apply the fallback per pod, before sum by (model).
3. Artifact is updated to head: final queries, the two fixes (f4fdb09, 6f51c1c), the script, and the before/after results.

I also rewrote my earlier review comments on this PR in English, following the new public-repository rule in AGENTS.md.

Description, verification (script output on head 2ccf70e) and artifact updated as requested.

[Copilot]

Copilot review overview

🟢 Approval recommended

Only a minor documentation nit remains; no blocking issues were identified.

Review effort: Lite
Findings: None

[hhk7734]

The dashboard was removed in #191