ACP token usage is lost on prompt cancel/timeout — timed-out runs report total_tokens=0

[EYH0602]

Title: ACP token usage is lost on prompt cancel/timeout — timed-out runs report total_tokens=0

Summary

For ACP agents, per-trial token usage is captured only from the return value of
session/prompt. When a prompt is cancelled or hits a hard/idle time cap, that request
never returns a PromptResult, so usage is never recorded and the trial reports
total_tokens=0 with usage_source=unavailable — even though the agent demonstrably
consumed tokens (a completed run of the same task reports millions).

This is not "imprecise" telemetry; it is zero telemetry on the exact runs (timeouts)
where a token count matters most for budget accounting.

Version

benchflow==0.6.5 (latest release as of filing).

Root cause

Usage is bound to prompt completion:

• ACPClient.prompt() awaits session/prompt, then records usage only after it returns:

  # benchflow/acp/client.py:407
  prompt_result = PromptResult.model_validate(result)
  self._session.stop_reason = StopReason(prompt_result.stop_reason)
  self._session.record_prompt_usage(getattr(prompt_result, "usage", None))

• On timeout the run goes through ACPClient.cancel() (acp/client.py:410), which sends
  session/cancel but never reaches record_prompt_usage.
• ACPSession.handle_update() — the session/update streaming handler
  (acp/session.py:273+) — does not inspect usage/tokens, so no incremental snapshot
  exists to fall back on.
• Result: usage_snapshots stays empty → latest_usage_totals() returns None
  (acp/session.py:242) → the trial emits usage_unavailable()
  (usage_tracking.py:70): total_tokens=0, usage_source=unavailable.

Reproduction

1. Run any ACP agent (e.g. claude-agent-acp) on a task with a hard/idle time cap short
   enough that the agent is still mid-prompt at cutoff.
2. The trial is cancelled at the cap.
3. Observe the result metrics: total_tokens=0, usage_source=unavailable, despite a
   completed run of the same task reporting real token counts.

Impact

• Any budget/cost accounting that depends on ACP token counts is unusable for timed-out
  trials — and timeouts are common in long-running / agentic tasks.
• Multi-turn ACP sessions partially mitigate (each completed prompt() leaves a
  cumulative snapshot, so only the in-flight turn's tokens are lost), but a single-prompt
  trial that times out loses everything.

Suggested fix (either or both)

1. Incremental capture from the stream. If the ACP agent emits usage in
   session/update notifications, record it in handle_update() so a snapshot exists
   before any cancel.
2. Capture-on-cancel. In the timeout/cancel path, read whatever partial usage is
   available (e.g. from the last session/update, or a usage field on the cancel
   acknowledgement if the protocol provides one) and append a snapshot before tearing down.

Even a best-effort partial count with a usage_source marking it as partial/at-cancel
would be a large improvement over 0.

Related

Prior ACP-usage work that fixed the completed-prompt path but not this edge case:
#249, #250, #183.

[Keesan12]

Zero is the one value BenchFlow knows is false here.

Usage accounting should not depend on session/prompt returning. Capture incremental usage events in a run ledger as they arrive, settle from the final result when available, and report a lower bound plus incomplete_due_to_timeout when it is not. That preserves honesty without pretending every provider exposes identical cancellation data.

This is exactly why MartinLoop keeps cost evidence outside the agent’s terminal return; we’ve now passed 10,000+ npm downloads testing governed runs. If you have a redacted timed-out ACP trace, I’ll compare the two accounting paths and send back where the lower bound diverges.

[bingran-you]

Daily scan triage (2026-07-22): confirmed this is repo-owned and currently has no linked fix PR.

Current main still has the relevant ACP usage/cancel paths in this repo: src/benchflow/acp/client.py records prompt usage only after session/prompt returns, ACPClient.cancel() sends session/cancel, and src/benchflow/acp/session.py owns the usage snapshot fallback consumed by rollout metrics. That makes this a BenchFlow ACP/rollout accounting fix rather than an off-repo task or provider-only issue.

Next safe step is a focused PR with a regression for timeout/cancelled ACP prompts, then best-effort partial usage capture from session/update and/or cancel acknowledgement where the ACP agent exposes it. Labels/project updated to bug, P1, status:ready, area:rollout, and the tasks project.