MAF-21239: feat(deploy): chart the request-observed KV cache hit per model

[hhk7734]

JIRA: MAF-21239
Artifact: MAF-21239 KV cache hit dashboard panels

Summary

This PR adds a "KV cache hit (request-observed)" row to the mif-aigateway dashboard. It shows, per model, how many input tokens the engines served from their KV cache, how much of the traffic that number covers, and how close the gateway's KV index prediction came. The panels read the per-request histograms that the Heimdall AI Gateway exports from its next release, and fall back to the older counters on gateways from v0.15.0 to v0.17.0.

Metrics used

Metric	Records
aigateway_engine_input_tokens	the engine's input token count, for every request whose engine returned usage
aigateway_engine_cached_tokens	the input tokens the engine served from its KV cache, for requests whose engine response included cached_tokens
aigateway_predicted_cached_tokens	the input tokens the gateway's KV index held as cached on the picked prefill worker, read before dispatch

All three are histograms with only the model label, so the row has no per-endpoint breakdown.

Panels

id	Panel	Series
84	KV cache hit ratio by model	measured engine_cached_sum / engine_input_sum, predicted predicted_sum / engine_input_sum
85	Cached-token coverage by model	engine engine_cached_count / engine_input_count, prediction predicted_count / engine_cached_count
86	Engine input, cached and predicted tokens/sec by model	the three _sum rates
87	Cached tokens per request p50/p95 by model	histogram_quantile over engine_cached_tokens_bucket

The measured hit ratio is a lower bound that is exact at 100% engine coverage, because a request without cached_tokens still adds its input tokens. The panel descriptions and the website docs say how to read the ratio with its coverage, and how it differs from the engine's own aggregate gauge.

During a rolling update, each gateway pod exports either the histograms or the older counters. The measured series take whichever set a pod has with or before summing by model. The predicted series and the per-request distribution need the histograms, and the predicted ratio divides only by newer pods' input tokens.

The Tokenizer cache and Odin InferenceService rows move down by 17 grid units. #188 changes only panels 77 to 79, so the two PRs do not overlap.

Test plan (head 41081e9)

• hack/dashboards/verify_aigateway_kv_cache_hit.py scrapes four fake gateways (two on the histograms, two on the older counters) with a local Prometheus 3.15 and runs all nine panel targets. Every target returns the expected value.
• Removing the older-counter fallback from the measured hit ratio fails the check. Removing the or 0 term from the engine coverage also fails it, because a model whose older gateway never saw cached_tokens disappears instead of showing 0%.
• make helm-lint and helm template pass, and the rendered ConfigMap contains the new queries.
• npm run build in website passes.
• Send a controlled cache-miss request and a cache-hit request, and check that the dashboard values match cached_tokens in the final responses.
• After merge, deploy the chart built by CI/CD to the target environment and check that the panels load in Grafana.

🤖 Generated with Claude Code

[hhk7734]

The dashboard was removed in #191