fix(metrics): normalize doc URL host in DocRetrievalRate [preventive] - #218
Draft
jessie1111101 wants to merge 1 commit into
Draft
fix(metrics): normalize doc URL host in DocRetrievalRate [preventive]#218jessie1111101 wants to merge 1 commit into
jessie1111101 wants to merge 1 commit into
Conversation
jessie1111101
marked this pull request as draft
July 22, 2026 17:45
Match mapped doc URLs against retrieved URIs regardless of the docs.cloud.google.com vs cloud.google.com host prefix, plus #fragment and trailing-slash differences. The Cloud docs site is served under both hosts and task mappings/retrieved URIs disagree on the prefix, so an exact substring match spuriously scored 0.0 even when docs were present. Mirrors the fix into the legacy pkg/ evaluator (run by the kube-agents harness today) and adds cross-host + fragment tests.
jessie1111101
force-pushed
the
fix/doc-retrieval-rate-url-normalization
branch
from
July 28, 2026 17:08
dc4df93 to
c98aff5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
DocRetrievalRatematches each mapped doc'surlagainst the JSON-serializedtrajectory via an exact, case-insensitive substring check. It hardcodes no host —
it just uses whatever the task mapping carries.
The latent inconsistency: the GKE Cloud docs site is served under both
cloud.google.comanddocs.cloud.google.com, and task mappings and retrieveddoc URIs disagree on the
docs.prefix per-doc, in both directions (verified:task.yaml files mix both hosts;
gke-mcp.logretrieved URIs mix both too). So anexact substring match can miss when the two sides differ on a page they otherwise
agree on.
Why this is preventive, not a confirmed fix
Every
DocRetrievalRate: 0.0we investigated for b/527099071 is explained by theprimary root cause — the retrieved docs never reach the outer trajectory at all
(the nested-MCP
generate_manifestreturns only the manifest YAML; docs live onlyin gke-mcp server logs). We have not observed a case where a same-page doc was
present in the trajectory yet scored 0 purely due to the host prefix. This change
guards against that once docs do start reaching the trajectory (post gke-mcp /
kube-agents fix, or via direct DK tool calls in kube-agents).
Change
Add
_normalize_doc_url: lowercase, collapsedocs.cloud.google.com->cloud.google.com, drop any#fragment, trim a trailing slash. Scheme is kept soshort URLs stay specific. Each trajectory step is collapsed the same way, so
matching is symmetric regardless of which side carries the prefix.
Applied to
devops_bench/metrics/grounding.py(canonical) and mirrored intopkg/evaluator/evaluate.py(the evaluator the kube-agents harness runs today).Tests
#fragmenttolerance.