Skip to content

Commit f02b89c

Browse files
Yuhan Gaocopybara-github
authored andcommitted
feat(tools): measure model consultation escalations, tokens, and cost
Each successful `ModelConsultTool` result reports the advisor tokens and latency of that one consultation, but nothing adds them up across turns and models or sets them against the executor's own usage, so there is no way to tell whether escalating to a stronger model pays off. Add `ModelConsultObservabilityPlugin`, exported from `google.adk.tools.model_consult` and `google.adk.plugins`. For every invocation it counts `ModelConsultTool` calls by outcome, advisor and executor tokens per model, advisor and end-to-end latency, the escalation rate, and an estimated cost when a price table is supplied. Each invocation ends with a `ModelConsultReport` passed to an optional sync or async `on_report` hook. The plugin also keeps cumulative `ModelConsultStats`, which `summary()` formats and `reset()` returns and clears in one step, so exporters can publish fixed windows without gaps. Closing a runner leaves runs in flight and the totals intact, so one instance can serve several runners at once, and the plugin tracks at most 10,000 unfinished invocations. The plugin works alongside `ForceEscalationPlugin` in either registration order and does not count the gated calls that plugin refuses. In live sessions it counts consultations but no executor calls. `ModelConsultReport`, `ModelConsultStats`, and `ModelConsultUsage` are exported from `google.adk.tools.model_consult`. Also add the developer unit guide under `docs/guides/tools/model_consult/observability_plugin/`. Co-authored-by: Yuhan Gao <gaoyuhan@google.com> PiperOrigin-RevId: 1001085702
1 parent cb54727 commit f02b89c

7 files changed

Lines changed: 3704 additions & 0 deletions

File tree

‎docs/guides/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,7 @@ This directory contains specific developer guides for the ADK Python implementat
103103

104104
### Plugins
105105
* [ForceEscalationPlugin](tools/model_consult/force_escalation_plugin/index.md) - Enforcing deterministic first-turn, repeated-tool-call, and pre-tool gating escalations to ModelConsultTool.
106+
* [ModelConsultObservabilityPlugin](tools/model_consult/observability_plugin/index.md) - Measuring ModelConsultTool escalations, advisor and executor tokens, latency, and estimated cost for every invocation.
106107
* [ReflectAndRetryModelPlugin](plugins/reflect_retry_model_plugin/index.md) - Self-healing, concurrent-safe error recovery for model failures.
107108
* [ReflectAndRetryToolPlugin](plugins/reflect_retry_tool_plugin/index.md) - Self-healing, concurrent-safe error recovery for tool failures.
108109
* [ToolCallIntegrityPlugin](plugins/tool_call_integrity_plugin/index.md) - Optional HMAC integrity check for stored function call arguments.

‎docs/guides/tools/model_consult/observability_plugin/index.md‎

Lines changed: 236 additions & 0 deletions
Large diffs are not rendered by default.

‎src/google/adk/plugins/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222

2323
if TYPE_CHECKING:
2424
from ..tools.model_consult._force_escalation_plugin import ForceEscalationPlugin
25+
from ..tools.model_consult._observability_plugin import ModelConsultObservabilityPlugin
2526
from ._reflect_retry_model_plugin import ReflectAndRetryModelPlugin
2627
from ._tool_call_integrity_plugin import ToolCallIntegrityError
2728
from ._tool_call_integrity_plugin import ToolCallIntegrityPlugin
@@ -34,6 +35,7 @@
3435
"DebugLoggingPlugin",
3536
"ForceEscalationPlugin",
3637
"LoggingPlugin",
38+
"ModelConsultObservabilityPlugin",
3739
"PluginManager",
3840
"ReflectAndRetryModelPlugin",
3941
"ReflectAndRetryToolPlugin",
@@ -45,6 +47,9 @@
4547
"DebugLoggingPlugin": "debug_logging_plugin",
4648
"ForceEscalationPlugin": "..tools.model_consult._force_escalation_plugin",
4749
"LoggingPlugin": "logging_plugin",
50+
"ModelConsultObservabilityPlugin": (
51+
"..tools.model_consult._observability_plugin"
52+
),
4853
"ReflectAndRetryModelPlugin": "_reflect_retry_model_plugin",
4954
"ReflectAndRetryToolPlugin": "reflect_retry_tool_plugin",
5055
"ToolCallIntegrityError": "_tool_call_integrity_plugin",

‎src/google/adk/tools/model_consult/__init__.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,17 @@
1717
from ._context import ModelConsultContextConfig
1818
from ._force_escalation_plugin import ForceEscalationPlugin
1919
from ._model_consult_tool import ModelConsultTool
20+
from ._observability_plugin import ModelConsultObservabilityPlugin
21+
from ._observability_plugin import ModelConsultReport
22+
from ._observability_plugin import ModelConsultStats
23+
from ._observability_plugin import ModelConsultUsage
2024

2125
__all__ = [
2226
'ForceEscalationPlugin',
2327
'ModelConsultContextConfig',
28+
'ModelConsultObservabilityPlugin',
29+
'ModelConsultReport',
30+
'ModelConsultStats',
2431
'ModelConsultTool',
32+
'ModelConsultUsage',
2533
]

0 commit comments

Comments
 (0)