Repository navigation
Commit f02b89c
feat(tools): measure model consultation escalations, tokens, and cost
Each successful `ModelConsultTool` result reports the advisor tokens and latency of that one consultation, but nothing adds them up across turns and models or sets them against the executor's own usage, so there is no way to tell whether escalating to a stronger model pays off.
Add `ModelConsultObservabilityPlugin`, exported from `google.adk.tools.model_consult` and `google.adk.plugins`. For every invocation it counts `ModelConsultTool` calls by outcome, advisor and executor tokens per model, advisor and end-to-end latency, the escalation rate, and an estimated cost when a price table is supplied. Each invocation ends with a `ModelConsultReport` passed to an optional sync or async `on_report` hook. The plugin also keeps cumulative `ModelConsultStats`, which `summary()` formats and `reset()` returns and clears in one step, so exporters can publish fixed windows without gaps. Closing a runner leaves runs in flight and the totals intact, so one instance can serve several runners at once, and the plugin tracks at most 10,000 unfinished invocations.
The plugin works alongside `ForceEscalationPlugin` in either registration order and does not count the gated calls that plugin refuses. In live sessions it counts consultations but no executor calls.
`ModelConsultReport`, `ModelConsultStats`, and `ModelConsultUsage` are exported from `google.adk.tools.model_consult`. Also add the developer unit guide under `docs/guides/tools/model_consult/observability_plugin/`.
Co-authored-by: Yuhan Gao <gaoyuhan@google.com>
PiperOrigin-RevId: 10010857021 parent cb54727 commit f02b89c
7 files changed
Lines changed: 3704 additions & 0 deletions
File tree
- docs/guides
- tools/model_consult/observability_plugin
- src/google/adk
- plugins
- tools/model_consult
- tests/unittests/tools/model_consult
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
| 106 | + | |
106 | 107 | | |
107 | 108 | | |
108 | 109 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
| |||
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
| 38 | + | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
| |||
45 | 47 | | |
46 | 48 | | |
47 | 49 | | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
48 | 53 | | |
49 | 54 | | |
50 | 55 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
20 | 24 | | |
21 | 25 | | |
22 | 26 | | |
23 | 27 | | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
24 | 31 | | |
| 32 | + | |
25 | 33 | | |
0 commit comments