Skip to content

fix(metrics): normalize doc URL host in DocRetrievalRate [preventive] - #218

Draft
jessie1111101 wants to merge 1 commit into
mainfrom
fix/doc-retrieval-rate-url-normalization
Draft

fix(metrics): normalize doc URL host in DocRetrievalRate [preventive]#218
jessie1111101 wants to merge 1 commit into
mainfrom
fix/doc-retrieval-rate-url-normalization

Conversation

@jessie1111101

@jessie1111101 jessie1111101 commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Status: draft / preventive hardening — not a fix for an observed failure.
Held until a real kube-agents run gives us a trajectory that actually contains
retrieved doc URIs, so we can confirm whether the host prefix genuinely
mismatches the task mappings for the same page. If it does, this fixes a
confirmed bug; if hosts line up, we may not need it.

Context

DocRetrievalRate matches each mapped doc's url against the JSON-serialized
trajectory via an exact, case-insensitive substring check. It hardcodes no host —
it just uses whatever the task mapping carries.

The latent inconsistency: the GKE Cloud docs site is served under both
cloud.google.com and docs.cloud.google.com, and task mappings and retrieved
doc URIs disagree on the docs. prefix per-doc, in both directions (verified:
task.yaml files mix both hosts; gke-mcp.log retrieved URIs mix both too). So an
exact substring match can miss when the two sides differ on a page they otherwise
agree on.

Why this is preventive, not a confirmed fix

Every DocRetrievalRate: 0.0 we investigated for b/527099071 is explained by the
primary root cause — the retrieved docs never reach the outer trajectory at all
(the nested-MCP generate_manifest returns only the manifest YAML; docs live only
in gke-mcp server logs). We have not observed a case where a same-page doc was
present in the trajectory yet scored 0 purely due to the host prefix. This change
guards against that once docs do start reaching the trajectory (post gke-mcp /
kube-agents fix, or via direct DK tool calls in kube-agents).

Change

Add _normalize_doc_url: lowercase, collapse docs.cloud.google.com ->
cloud.google.com, drop any #fragment, trim a trailing slash. Scheme is kept so
short URLs stay specific. Each trajectory step is collapsed the same way, so
matching is symmetric regardless of which side carries the prefix.

Applied to devops_bench/metrics/grounding.py (canonical) and mirrored into
pkg/evaluator/evaluate.py (the evaluator the kube-agents harness runs today).

Tests

  • New: cross-host-prefix matching and #fragment tolerance.
  • All 14 grounding tests pass; canonical files ruff-clean.
  • Behavioral check is synthetic (constructed trajectory), not from an observed run.

@jessie1111101
jessie1111101 marked this pull request as draft July 22, 2026 17:45
@jessie1111101 jessie1111101 changed the title fix(metrics): normalize doc URL host in DocRetrievalRate fix(metrics): normalize doc URL host in DocRetrievalRate [preventive] Jul 22, 2026
Match mapped doc URLs against retrieved URIs regardless of the
docs.cloud.google.com vs cloud.google.com host prefix, plus #fragment
and trailing-slash differences. The Cloud docs site is served under both
hosts and task mappings/retrieved URIs disagree on the prefix, so an
exact substring match spuriously scored 0.0 even when docs were present.

Mirrors the fix into the legacy pkg/ evaluator (run by the kube-agents
harness today) and adds cross-host + fragment tests.
@jessie1111101
jessie1111101 force-pushed the fix/doc-retrieval-rate-url-normalization branch from dc4df93 to c98aff5 Compare July 28, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant