diff --git a/deploy/helm/AGENTS.md b/deploy/helm/AGENTS.md
index 63f20e40..af631b49 100644
--- a/deploy/helm/AGENTS.md
+++ b/deploy/helm/AGENTS.md
@@ -45,6 +45,7 @@ Instead, the regular `templates/minio/init-job.yaml` Job (not a hook) provisions
 
 - AIGateway dashboard coverage audits must include first-party metrics from gateway request metrics, gateway exposition, and sidecar metrics. Use `rate` for counters, guard derived ratios against zero denominators, and aggregate replica-local series deliberately before grouping by model or endpoint.
 - AIGateway metrics carry no `role` label; endpoint-role filtering joins Kubernetes metadata instead. `kube_pod_labels` exposes `label_mif_moreh_io_role` only because `prometheus-stack.kube-state-metrics.metricLabelsAllowlist` allowlists `pods=[mif.moreh.io/role]` — keep that allowlist and the dashboards in sync. Once a resource has any allowlist entry, KSM emits `kube_<resource>_labels` for **every** object of that resource (allowlisted labels appear only where set), so `=~".*"` keeps label-less objects while `=~".+"` or a specific value drops them.
+- A per-model ratio whose numerator series appears only after its first event (e.g. `aigateway_cached_tokens_reports_total{result="reported"}`) must fall back to zero per model — `(sum by (model) (rate(x{result="reported"}[..])) or 0 * sum by (model) (rate(x[..]))) / (sum by (model) (rate(x[..])) > 0)` — or a model with no reports disappears instead of showing 0%. `hack/dashboards/verify_aigateway_kv_cache_hit.py` checks the request-observed KV cache hit panels against fake gateways; run it after editing those panels.
 - A Grafana template variable built from a PromQL join (anything that is not a plain series selector) must use `query_result(<expr>)` plus a `regex` such as `/endpoint="([^"]+)"/`; `label_values(<expr>, <label>)` resolves through the series/label-values API and only accepts selectors.
 
 ## Alert provisioning
diff --git a/deploy/helm/moai-inference-framework/files/dashboards/mif-aigateway.json b/deploy/helm/moai-inference-framework/files/dashboards/mif-aigateway.json
index 21102c1e..e5df8475 100644
--- a/deploy/helm/moai-inference-framework/files/dashboards/mif-aigateway.json
+++ b/deploy/helm/moai-inference-framework/files/dashboards/mif-aigateway.json
@@ -1574,6 +1574,280 @@
         "x": 0,
         "y": 74
       },
+      "id": 83,
+      "panels": [],
+      "title": "KV cache hit (request-observed)",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "description": "measured: cached tokens / input tokens, both as the engine counted them. A request whose engine response has no cached_tokens adds input tokens and no cached tokens, so this is a lower bound that is exact when coverage is 100%. predicted: the input tokens that the gateway's KV index held as cached on the picked worker, read before dispatch, over the same input tokens. A gap between the two shows how far the KV index is from the engine's cache; check the prediction coverage first. These values come from each request's final engine response. They are not the engine's own aggregate prefix cache gauge (vllm:gpu_prefix_cache_hit_rate, sglang:cache_hit_rate), which mixes all traffic on one worker.",
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "custom": {
+            "drawStyle": "line",
+            "lineWidth": 1,
+            "fillOpacity": 0,
+            "showPoints": "auto"
+          },
+          "color": {
+            "mode": "palette-classic"
+          },
+          "min": 0,
+          "max": 1
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 75
+      },
+      "id": 84,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (model) (rate(aigateway_engine_cached_tokens_sum{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval]) or rate(aigateway_prompt_tokens_observed_total{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\", source=\"cache\"}[$__rate_interval])) / (sum by (model) (rate(aigateway_engine_input_tokens_sum{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval]) or sum without (source) (rate(aigateway_prompt_tokens_observed_total{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval]))) > 0)",
+          "legendFormat": "{{model}} measured",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (model) (rate(aigateway_predicted_cached_tokens_sum{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval])) / (sum by (model) (rate(aigateway_engine_input_tokens_sum{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval])) > 0)",
+          "legendFormat": "{{model}} predicted",
+          "refId": "B"
+        }
+      ],
+      "title": "KV cache hit ratio by model",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "description": "engine: the share of requests whose engine response included cached_tokens. Below 100% on vLLM usually means a worker runs without --enable-prompt-tokens-details; MoAI Engine and SGLang always include it. prediction: the share of those requests that also had a scheduler prediction. A PD request that fell back to E2E has cached tokens and no prediction. A request that the client cancels mid-stream is not counted.",
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "custom": {
+            "drawStyle": "line",
+            "lineWidth": 1,
+            "fillOpacity": 0,
+            "showPoints": "auto"
+          },
+          "color": {
+            "mode": "palette-classic"
+          },
+          "min": 0,
+          "max": 1
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 75
+      },
+      "id": 85,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (model) (rate(aigateway_engine_cached_tokens_count{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval]) or rate(aigateway_cached_tokens_reports_total{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\", result=\"reported\"}[$__rate_interval]) or 0 * sum without (result) (rate(aigateway_cached_tokens_reports_total{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval]))) / (sum by (model) (rate(aigateway_engine_input_tokens_count{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval]) or sum without (result) (rate(aigateway_cached_tokens_reports_total{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval]))) > 0)",
+          "legendFormat": "{{model}} engine",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (model) (rate(aigateway_predicted_cached_tokens_count{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval])) / (sum by (model) (rate(aigateway_engine_cached_tokens_count{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval])) > 0)",
+          "legendFormat": "{{model}} prediction",
+          "refId": "B"
+        }
+      ],
+      "title": "Cached-token coverage by model",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "description": "Tokens per second behind the hit ratio panel: the engine's input tokens, the input tokens it served from its KV cache, and the input tokens the scheduler predicted as cached. The engine's input count can differ from Input tokens/sec, which the gateway tokenizer counts, when the engine expands multimodal input.",
+      "fieldConfig": {
+        "defaults": {
+          "unit": "cps",
+          "custom": {
+            "drawStyle": "line",
+            "lineWidth": 1,
+            "fillOpacity": 0,
+            "showPoints": "auto"
+          },
+          "color": {
+            "mode": "palette-classic"
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 83
+      },
+      "id": 86,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (model) (rate(aigateway_engine_input_tokens_sum{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval]) or sum without (source) (rate(aigateway_prompt_tokens_observed_total{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval])))",
+          "legendFormat": "{{model}} input",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (model) (rate(aigateway_engine_cached_tokens_sum{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval]) or rate(aigateway_prompt_tokens_observed_total{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\", source=\"cache\"}[$__rate_interval]))",
+          "legendFormat": "{{model}} cached",
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (model) (rate(aigateway_predicted_cached_tokens_sum{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval]))",
+          "legendFormat": "{{model}} predicted",
+          "refId": "C"
+        }
+      ],
+      "title": "Engine input, cached and predicted tokens/sec by model",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "description": "Distribution of the input tokens the engine served from its KV cache per request, over the requests whose engine response included cached_tokens. A high p95 with a low hit ratio means a few long prefixes are reused while most requests miss.",
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "custom": {
+            "drawStyle": "line",
+            "lineWidth": 1,
+            "fillOpacity": 0,
+            "showPoints": "auto"
+          },
+          "color": {
+            "mode": "palette-classic"
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 83
+      },
+      "id": 87,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.50, sum by (le, model) (rate(aigateway_engine_cached_tokens_bucket{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval])))",
+          "legendFormat": "{{model}} p50",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.95, sum by (le, model) (rate(aigateway_engine_cached_tokens_bucket{namespace=~\"$namespace\", aigateway=~\"$aigateway\", model=~\"$model\"}[$__rate_interval])))",
+          "legendFormat": "{{model}} p95",
+          "refId": "B"
+        }
+      ],
+      "title": "Cached tokens per request p50/p95 by model",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 91
+      },
       "id": 76,
       "panels": [],
       "title": "Tokenizer cache",
@@ -1605,7 +1879,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 75
+        "y": 92
       },
       "id": 77,
       "options": {
@@ -1658,7 +1932,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 75
+        "y": 92
       },
       "id": 78,
       "options": {
@@ -1720,7 +1994,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 83
+        "y": 100
       },
       "id": 79,
       "options": {
@@ -1773,7 +2047,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 83
+        "y": 100
       },
       "id": 80,
       "options": {
@@ -1826,7 +2100,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 91
+        "y": 108
       },
       "id": 81,
       "options": {
@@ -1879,7 +2153,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 91
+        "y": 108
       },
       "id": 82,
       "options": {
@@ -1913,7 +2187,7 @@
         "h": 1,
         "w": 24,
         "x": 0,
-        "y": 99
+        "y": 116
       },
       "id": 63,
       "panels": [],
@@ -1947,7 +2221,7 @@
         "h": 4,
         "w": 8,
         "x": 0,
-        "y": 100
+        "y": 117
       },
       "id": 64,
       "options": {
@@ -2005,7 +2279,7 @@
         "h": 4,
         "w": 8,
         "x": 8,
-        "y": 100
+        "y": 117
       },
       "id": 65,
       "options": {
@@ -2070,7 +2344,7 @@
         "h": 4,
         "w": 8,
         "x": 16,
-        "y": 100
+        "y": 117
       },
       "id": 66,
       "options": {
@@ -2125,7 +2399,7 @@
         "h": 8,
         "w": 24,
         "x": 0,
-        "y": 104
+        "y": 121
       },
       "id": 72,
       "options": {
@@ -2235,7 +2509,7 @@
         "h": 8,
         "w": 24,
         "x": 0,
-        "y": 112
+        "y": 129
       },
       "id": 67,
       "options": {
@@ -2342,7 +2616,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 120
+        "y": 137
       },
       "id": 68,
       "options": {
@@ -2394,7 +2668,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 120
+        "y": 137
       },
       "id": 69,
       "options": {
@@ -2448,7 +2722,7 @@
         "h": 8,
         "w": 24,
         "x": 0,
-        "y": 128
+        "y": 145
       },
       "id": 73,
       "options": {
@@ -2502,7 +2776,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 136
+        "y": 153
       },
       "id": 74,
       "options": {
@@ -2556,7 +2830,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 136
+        "y": 153
       },
       "id": 75,
       "options": {
diff --git a/hack/dashboards/verify_aigateway_kv_cache_hit.py b/hack/dashboards/verify_aigateway_kv_cache_hit.py
new file mode 100644
index 00000000..8c4ea953
--- /dev/null
+++ b/hack/dashboards/verify_aigateway_kv_cache_hit.py
@@ -0,0 +1,231 @@
+#!/usr/bin/env python3
+"""Run the AIGateway dashboard's request-observed KV cache hit panel queries
+against a local Prometheus fed by fake gateways, and check the values each
+panel reports.
+
+The fake gateways model a rolling update from the old counters
+(aigateway_prompt_tokens_observed_total, aigateway_cached_tokens_reports_total)
+to the per-request histograms (aigateway_engine_input_tokens,
+aigateway_engine_cached_tokens, aigateway_predicted_cached_tokens). Per
+second:
+
+- model A, new pod: three requests of 100 input tokens. Two had 64 cached
+  tokens and a 48-token prediction; one had no cached_tokens.
+- model A, old pod: 64 cached and 36 prefilled tokens from one reported
+  request, plus one absent request.
+- model B, old pod: two absent requests only, like a vLLM worker without
+  --enable-prompt-tokens-details. The old gateway never creates the token
+  series for it.
+- model C, new pod: one request of 50 input tokens with no cached_tokens.
+  The new gateway still creates the cached series, at count 0.
+
+Usage:
+    python3 hack/dashboards/verify_aigateway_kv_cache_hit.py --prometheus /path/to/prometheus
+
+The Prometheus binary comes from https://github.com/prometheus/prometheus/releases.
+Exits non-zero when a panel query returns a value other than the expected one.
+"""
+
+import argparse
+import http.server
+import json
+import math
+import pathlib
+import socket
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+import urllib.parse
+import urllib.request
+
+DASHBOARD = (
+    pathlib.Path(__file__).resolve().parents[2]
+    / "deploy/helm/moai-inference-framework/files/dashboards/mif-aigateway.json"
+)
+
+LABELS = 'namespace="ns",aigateway="gw"'
+
+# The gateway's token buckets: 1 to 262144, four times apart.
+BUCKETS = [4**i for i in range(10)]
+
+
+def histogram(name, model, observations):
+    """Series of one histogram, as (series, value per second).
+
+    `observations` lists (count per second, tokens per observation).
+    """
+    labels = f'model="{model}"'
+    series = []
+    for le in BUCKETS:
+        count = sum(c for c, v in observations if v <= le)
+        series.append((f'{name}_bucket{{{labels},le="{le}"}}', count))
+    total = sum(c for c, _ in observations)
+    series.append((f'{name}_bucket{{{labels},le="+Inf"}}', total))
+    series.append((f"{name}_sum{{{labels}}}", sum(c * v for c, v in observations)))
+    series.append((f"{name}_count{{{labels}}}", total))
+    return series
+
+
+PODS = {
+    "new-a": histogram("aigateway_engine_input_tokens", "A", [(3, 100)])
+    + histogram("aigateway_engine_cached_tokens", "A", [(2, 64)])
+    + histogram("aigateway_predicted_cached_tokens", "A", [(2, 48)]),
+    "old-a": [
+        ('aigateway_prompt_tokens_observed_total{model="A",source="cache"}', 64),
+        ('aigateway_prompt_tokens_observed_total{model="A",source="prefill"}', 36),
+        ('aigateway_cached_tokens_reports_total{model="A",result="reported"}', 1),
+        ('aigateway_cached_tokens_reports_total{model="A",result="absent"}', 1),
+    ],
+    "old-b": [
+        ('aigateway_cached_tokens_reports_total{model="B",result="absent"}', 2),
+    ],
+    "new-c": histogram("aigateway_engine_input_tokens", "C", [(1, 50)])
+    + histogram("aigateway_engine_cached_tokens", "C", []),
+}
+
+NAN = float("nan")
+
+# Expected value per (panel id, target refId), keyed by model. A missing model
+# means the query must return no series for it.
+EXPECTED = {
+    # Cached over input tokens across both pods of model A. Model B exported no
+    # tokens, so it has no ratio instead of a false 0%.
+    (84, "A"): {"A": (128 + 64) / (300 + 100), "C": 0.0},
+    # Only the new pod predicts, over its own input tokens.
+    (84, "B"): {"A": 96 / 300},
+    # Requests with cached tokens over all requests, either version. Model B
+    # has none and shows 0%, not a missing series.
+    (85, "A"): {"A": (2 + 1) / (3 + 2), "B": 0.0, "C": 0.0},
+    # Every request of model A with cached tokens on the new pod was predicted.
+    (85, "B"): {"A": 1.0},
+    (86, "A"): {"A": 300 + 100, "C": 50},
+    (86, "B"): {"A": 128 + 64, "C": 0.0},
+    (86, "C"): {"A": 96},
+    # Every cached observation of model A is 64 tokens, in the (16, 64] bucket.
+    (87, "A"): {"A": 16 + (64 - 16) * 0.50, "C": NAN},
+    (87, "B"): {"A": 16 + (64 - 16) * 0.95, "C": NAN},
+}
+
+RATE_WINDOW = "20s"
+
+
+def free_port():
+    with socket.socket() as s:
+        s.bind(("127.0.0.1", 0))
+        return s.getsockname()[1]
+
+
+def serve_pod(port, series, started):
+    class Handler(http.server.BaseHTTPRequestHandler):
+        def do_GET(self):
+            elapsed = time.time() - started
+            body = "".join(
+                f"{name.replace('{', '{' + LABELS + ',', 1)} {rate * elapsed}\n"
+                for name, rate in series
+            )
+            self.send_response(200)
+            self.send_header("Content-Type", "text/plain; version=0.0.4")
+            self.end_headers()
+            self.wfile.write(body.encode())
+
+        def log_message(self, *args):
+            pass
+
+    server = http.server.HTTPServer(("127.0.0.1", port), Handler)
+    threading.Thread(target=server.serve_forever, daemon=True).start()
+
+
+def query(base, expr):
+    url = f"{base}/api/v1/query?" + urllib.parse.urlencode({"query": expr})
+    with urllib.request.urlopen(url) as resp:
+        body = json.load(resp)
+    if body["status"] != "success":
+        raise RuntimeError(f"query failed: {body}")
+    return {r["metric"].get("model"): float(r["value"][1]) for r in body["data"]["result"]}
+
+
+def close(got, want):
+    if math.isnan(want):
+        return math.isnan(got)
+    return math.isclose(got, want, rel_tol=1e-3, abs_tol=1e-9)
+
+
+def panel_exprs():
+    dashboard = json.loads(DASHBOARD.read_text())
+    panels = {p["id"]: p for p in dashboard["panels"]}
+    for (panel_id, ref_id) in EXPECTED:
+        target = next(t for t in panels[panel_id]["targets"] if t["refId"] == ref_id)
+        expr = (
+            target["expr"]
+            .replace("$namespace", ".*")
+            .replace("$aigateway", ".*")
+            .replace("$model", ".*")
+            .replace("$__rate_interval", RATE_WINDOW)
+        )
+        yield (panel_id, ref_id), panels[panel_id]["title"], expr
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    parser.add_argument("--prometheus", default="prometheus", help="Prometheus binary")
+    args = parser.parse_args()
+
+    started = time.time()
+    targets = []
+    for series in PODS.values():
+        port = free_port()
+        serve_pod(port, series, started)
+        targets.append(f"127.0.0.1:{port}")
+
+    prom_port = free_port()
+    base = f"http://127.0.0.1:{prom_port}"
+    with tempfile.TemporaryDirectory() as tmp:
+        config = pathlib.Path(tmp, "prometheus.yml")
+        config.write_text(
+            "global: {scrape_interval: 1s}\n"
+            "scrape_configs:\n"
+            "  - job_name: fake-aigateway\n"
+            "    honor_labels: true\n"
+            f"    static_configs: [{{targets: {json.dumps(targets)}}}]\n"
+        )
+        prom = subprocess.Popen(
+            [
+                args.prometheus,
+                f"--config.file={config}",
+                f"--web.listen-address=127.0.0.1:{prom_port}",
+                f"--storage.tsdb.path={tmp}/data",
+            ],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+        )
+        try:
+            # Enough samples in every series for a full rate window.
+            deadline = time.time() + 120
+            while True:
+                try:
+                    counts = query(base, "min(count_over_time(up[1m]))")
+                    if counts and min(counts.values()) >= 25:
+                        break
+                except OSError:
+                    pass
+                if time.time() > deadline:
+                    sys.exit("Prometheus collected no samples within 120s")
+                time.sleep(1)
+
+            failed = False
+            for key, title, expr in panel_exprs():
+                got = query(base, expr)
+                want = EXPECTED[key]
+                ok = got.keys() == want.keys() and all(close(got[m], want[m]) for m in want)
+                failed |= not ok
+                print(f"{'ok  ' if ok else 'FAIL'} panel {key[0]} {key[1]} {title}: got {got}, want {want}")
+            sys.exit(1 if failed else 0)
+        finally:
+            prom.terminate()
+            prom.wait()
+
+
+if __name__ == "__main__":
+    main()
diff --git a/website/docs/operations/monitoring/metrics/index.mdx b/website/docs/operations/monitoring/metrics/index.mdx
index 8b541da4..bb13cc2f 100644
--- a/website/docs/operations/monitoring/metrics/index.mdx
+++ b/website/docs/operations/monitoring/metrics/index.mdx
@@ -133,6 +133,37 @@ This section reports the request traffic the Heimdall AI Gateway routes, both in
 - **ISL p50/p75/p95 by model**: Input sequence length distribution per model.
 - **OSL p50/p75/p95 by model**: Output sequence length distribution per model.
 
+### KV cache hit (request-observed)
+
+This section reports how many input tokens the inference engines actually served from their KV cache, per served model, and how close the gateway's own prediction came. The Heimdall AI Gateway reads `usage.prompt_tokens` and `usage.prompt_tokens_details.cached_tokens` from the final engine response of each request and exports them as per-request histograms. In a prefill-decode disaggregated deployment, the cached count comes from the prefill worker. The histograms carry only the `model` label, so the section has no per-endpoint breakdown. The gateway records them for `/v1/chat/completions`, `/v1/completions`, `/v1/messages`, and `/v1/responses`, streaming or not.
+
+| Metric | Records |
+| --- | --- |
+| `aigateway_engine_input_tokens` | The engine's input token count, for every request whose engine returned usage |
+| `aigateway_engine_cached_tokens` | The input tokens the engine served from its KV cache, for requests whose engine response included `cached_tokens` |
+| `aigateway_predicted_cached_tokens` | The input tokens the gateway's KV index held as cached on the worker picked to run prefill, read before the request was sent, for the same requests as `aigateway_engine_cached_tokens` |
+
+- **KV cache hit ratio by model**: `measured` is cached tokens divided by input tokens, both as the engine counted them. A request without `cached_tokens` adds input tokens and no cached tokens, so the ratio is exact only when coverage is 100% and a lower bound otherwise. `predicted` is the predicted cached tokens over the same input tokens.
+- **Cached-token coverage by model**: `engine` is the share of requests whose engine response included `cached_tokens`. `prediction` is the share of those requests that also had a prediction. Compare the two hit ratios only when prediction coverage is close to 100%.
+- **Engine input, cached and predicted tokens/sec by model**: The token rates behind the hit ratio panel.
+- **Cached tokens per request p50/p95 by model**: The distribution of cached tokens per request. A high p95 with a low hit ratio means a few long prefixes are reused while most requests miss.
+
+:::info
+vLLM returns `cached_tokens` only when the worker runs with `--enable-prompt-tokens-details`. MoAI Engine and SGLang always return it. If coverage stays below 100% for a model served by vLLM, check the engine arguments of its workers before you read the hit ratio. A request that the client cancels mid-stream is not counted in any panel of this section.
+:::
+
+Gateways that do not export these histograms yet export the older `aigateway_prompt_tokens_observed_total` and `aigateway_cached_tokens_reports_total` counters instead. The measured hit ratio, the engine coverage, and the input and cached token rates read those counters on such gateways, so the panels keep their values during a rolling update. The predicted series and the per-request distribution need the histograms.
+
+Three different signals are called a cache hit rate, and they answer different questions. Only this section shows the result of each request.
+
+| Signal | Source | Meaning |
+| --- | --- | --- |
+| Measured hit ratio (this section) | `cached_tokens` in each final engine response | The share of input tokens served from the KV cache, per model. Use it to judge whether prefix caching pays off. |
+| Predicted hit ratio (this section) | The gateway's KV index, read for the picked worker before the request is sent | A prediction, not a result. Its gap from the measured ratio shows how far the index is from the engine's cache. |
+| Engine aggregate hit rate | The engine's own gauge, such as `vllm:gpu_prefix_cache_hit_rate` or `sglang:cache_hit_rate` | The running hit rate of one worker across all of its traffic. It is not per request and not per model. |
+
+The dashboard's **Tokenizer cache** section, which follows this one, is unrelated to the KV cache. It shows the gateway's own tokenizer cache.
+
 ### Odin InferenceService
 
 :::info