BrowserDisconnectedError is reported as "You must use exactly one tool", so handleBrowserDisconnect() never runs

[lmorchard]

Summary

When a tool's execute() throws, the AI SDK records the error as a tool result rather than propagating the throw, and aiResponse.toolResults comes back empty. webAgent.ts reads empty toolResults as "the model called no tool" and throws ToolExecutionError("You must use exactly one tool."), discarding the original error's type.

For BrowserDisconnectedError this is expensive: it is exactly the error handleBrowserDisconnect() exists to recover from, and the instanceof check that would trigger recovery can never match. Pilo has complete, correct reconnect logic that is unreachable via this path.

Impact

Measured on published hundred.jsonl eval runs, build 7d0f381688f5394892eead1fe28fe4fd7ebf6a9f:

• 19 of 52 task failures (37%) — the single largest failure mode on that build.
• browser:reconnected fired 0 times across all 19, despite a real BrowserDisconnectedError in every one.
• Pass rate falls from 81% (159 attempts with no such event) to 8% (12 attempts where it happened 3×).

toolCallMalformed count	n	aborts claiming disconnect	pass rate
0	159	0 (0%)	81%
1	8	12%	50%
2	8	62%	38%
3	12	92%	8%

Concentrated on Coursera (7 failures) and Cambridge Dictionary (5) — enough to account for Coursera's entire deficit against browser-use in a 3-way comparison, where browser-use passes the same tasks on the same Bright Data endpoint (verified: the CDP secret keys are byte-identical, so this is not a provider difference).

The disconnects themselves are upstream — ~11% of attempts on Bright Data vs 0/930 on browserless with the same agent build. But the failure to recover is pilo's, and the recovery is already written.

Root cause

1. The browser layer throws BrowserDisconnectedError — packages/core/src/browser/playwrightBrowser.ts:729.

2. BrowserDisconnectedError extends RecoverableError, not BrowserException (packages/core/src/errors.ts:184), so webActionTools.ts:255's instanceof BrowserException catch correctly does not swallow it and line 287 re-throws. This part works as intended.

3. But the throw happens inside the AI SDK's tool execution, beneath streamText. The SDK catches it, records a tool result with output.type === 'error-text', and resolves normally — with toolResults empty. From the artifacts:

{
  "type": "tool-result",
  "toolName": "enter",
  "output": {
    "type": "error-text",
    "value": "BrowserDisconnectedError: Browser connection lost: locator.press: Target page, context or browser has been closed\nBrowser logs:\n\ninternal server error (brob)"
  }
}

4. packages/core/src/webAgent.ts:1296-1300 then treats that as a no-tool-call:

// Process tool results
if (!aiResponse?.toolResults?.length) {
  console.error("[WebAgent] No tools called in action generation");
  throw new ToolExecutionError(
    "You must use exactly one tool. Please use one of the available tools.",

This is the defect. toolResults.length === 0 conflates "the model called no tool" with "the tool the model called threw."

5. So the dispatch at webAgent.ts:738 can never match for this class:

if (error instanceof BrowserDisconnectedError) {
  await this.handleBrowserDisconnect(task, error, executionState);

6. addErrorFeedback() (webAgent.ts:889) then early-returns for ToolExecutionError:

IMPORTANT: Tool execution errors (ToolExecutionError) are NOT added as user messages because the error information is already present in the tool result output.

Sound in general, but false for the branch at line 1297 — which fires precisely because there is no tool result carrying that information.

7. The model does see the error-text in its message history, correctly concludes the browser is dead, and after 3–4 such turns aborts saying so. The agent's abort narration is accurate — it is the only component diagnosing this correctly. (Worth noting because downstream eval tooling had been classifying these as agent confabulation.)

Relationship to #637

#637 added system:debug_no_tool_call to separate prose-instead-of-tool from truncation. It records a third case neither hypothesis covers:

{'iterationId': 'pvgwucsK', 'finishReason': 'tool-calls', 'textLength': '0', 'textPreview': ''}

finishReason: 'tool-calls' with zero text — the model did emit a tool call. It was executed, and it threw. That diagnostic is what made this findable.

Suggested fix

In the !aiResponse?.toolResults?.length branch, distinguish the two cases before throwing: inspect the returned response.messages for a tool-result part with output.type === 'error-text' and re-throw the original typed error (minimally, throw BrowserDisconnectedError when the value matches) so the existing instanceof dispatch works. Fall back to "You must use exactly one tool" only when there genuinely was no tool call.

A tidier variant: have the tool wrappers use the SDK's typed tool-error channel so pilo never infers type from a string. Larger change; the narrow fix restores recovery on its own.

Separately worth deciding: whether addErrorFeedback should push a user message for the true no-tool-call case, since today that path gives the model no feedback at all.

What still needs confirming

Step 3 is inferred from artifacts, not read out of the SDK. The evidence is strong — the error-text tool result, finishReason: 'tool-calls' with empty text, and the ToolExecutionError firing — but nobody has confirmed against ai@^7.0.16 that toolResults excludes errored calls in all cases, or checked whether the SDK exposes a typed error channel that would make the fix cleaner. Please verify before settling on an approach.

Also unknown: whether the same swallowing hides other typed errors with their own recovery paths. BrowserDisconnectedError is the one with a measured cost.

Reproduction

Not yet run. Proposed:

1. Point pilo at a CDP endpoint you can kill (local chromedp/headless-shell or browserless), PILO_REQUIRE_CDP=true.
2. Start a multi-step task; once it's interacting with the page, kill the browser container.
3. Current: tool:execution:error = "You must use exactly one tool", no browser:reconnected, agent aborts within ~4 iterations blaming the browser. Fixed: browser:reconnected emitted, task restarts on the next endpoint.

A unit test is the better regression guard — stub the AI call to return toolResults: [] plus a response.messages entry containing an error-text output whose value is a BrowserDisconnectedError message, and assert handleBrowserDisconnect() is invoked. packages/core/test/webAgent.test.ts already stubs generation this way.

---

Source claims cited against mozilla/pilo@7446077e. Evidence: 19 task artifacts from scheduled-bu-pilo-tab-hundred runs 1785337200 / 1785423600, participant pilo-cli — public, no auth: https://storage.googleapis.com/pilo-public-eval-reports/reports/<run>/<task-id>/result.json. Full analysis: Mozilla-Ocho/pilo-evals-judge#122 and docs/dev-sessions/2026-07-31-1100-issue-122-per-site-gaps/research.md in that repo.

🤖 Investigated with Claude Code

[lmorchard]

Step 3 confirmed against the SDK — and the fix is cleaner than I proposed

The "what still needs confirming" section asked two things: whether toolResults really excludes errored tool calls, and whether the SDK exposes a typed error channel that would make the fix cleaner. Both are now answered — yes and yes.

Version note first. packages/core on current origin/main declares ai: ^7.0.26 and resolves to 7.0.26; that is what I read and ran, not the ^7.0.16 quoted in the issue. A stale node_modules in a long-lived checkout can hold 6.0.197, which is a materially different code path — check packages/core/node_modules/ai/package.json before trusting a local read.

Read out of the SDK

toolResults is a filter over content by part type:

// node_modules/ai/dist/index.js:4348
get toolResults() {
  return this.content.filter((part) => part.type === "tool-result");
}

A throw from execute() is caught in executeToolCall and returned as a different part type, carrying the original error object rather than a string:

// node_modules/ai/dist/index.js:2943-2949
} catch (error) {
  const toolError = {
    type: "tool-error",
    toolCallId, toolName, input,
    error,                      // <- the thrown instance, unmodified

dist/index.d.ts types this as error: unknown on StaticToolError / DynamicToolError, and the ContentPart union includes tool-error. So the part is present in content and excluded from toolResults by construction. Step 3 holds, and it is structural rather than incidental.

Confirmed by running it

MockLanguageModelV4 + simulateReadableStream, one identical model stream, only the tool body differs:

tool body	toolResults.length	content part types
returns "ok"	1	tool-call, tool-result
throws BrowserDisconnectedError	0	tool-call, tool-error

and on the throwing arm:

tool-error present   : true
  error is Error     : true
  constructor.name   : BrowserDisconnectedError
  instanceof BDE     : true

The A/B is the point: same stream both times, so the zero comes from the throw and nothing else. toolResults.length === 0 → pilo's branch fires → ToolExecutionError, exactly as the eval artifacts showed.

The typed channel exists, so drop the string matching

The issue suggested inspecting response.messages for output.type === 'error-text' and re-throwing BrowserDisconnectedError "when the value matches". I withdraw that. response.messages is the stringified, model-facing view; content is the typed, program-facing one, and it hands back the live instance. pilo already awaits five sibling promises at webAgent.ts:1271-78, so content just joins that list:

const toolError = content.find((p) => p.type === "tool-error");
if (toolError) throw toolError.error;   // instanceof dispatch at :766 now matches

No string matching, no inferring type from a message, and it generalises to every typed error instead of just this one.

Which channels carry the typed instance (all measured, not assumed):

channel	typed instance?
result.content	✅
steps[].content	✅
onStepFinish(step).content	✅
onToolExecutionEnd (toolOutput.type === 'tool-error')	✅
fullStream tool-error part	✅
onError	❌ never fires — 0 invocations

onError is the obvious first guess and it is the wrong one. Worth knowing before someone builds a fix on it.

On the issue's other open question

"Whether the same swallowing hides other typed errors with their own recovery paths" — structurally yes, all of them. The filter is by part type, so any error thrown from a tool body is excluded from toolResults regardless of class. In practice BrowserDisconnectedError is the only one that matters today, because BrowserException is caught inside the tool (webActionTools.ts:255) and never reaches the SDK boundary. So the narrow fix is sufficient right now — the typed re-throw above is what keeps it sufficient as more recoverable errors appear.

Two caveats

• finishReason in my probe is other on both arms, not the tool-calls the real artifacts show. That is a mock artifact, not a finding: the control arm (succeeding tool, same stream) also reports other. It is not evidence against the real value.
• One narrower path does stringify. Invalid dynamic tool calls at dist/index.js:5514 use getErrorMessage(toolCall.error). That is a different case — malformed input, not a throw from execute — but it means "always typed" is not universal.

Line drift

The issue cites 7446077e. On current origin/main the same code sits at webAgent.ts:1339 (the !toolResults?.length branch), :766 (the instanceof dispatch), and :2207 (handleBrowserDisconnect).

Probe script (drop in packages/core/, node ./probe.mjs)

import { streamText, tool } from "ai";
import { MockLanguageModelV4, simulateReadableStream } from "ai/test";
import { z } from "zod";

class BrowserDisconnectedError extends Error {
  constructor(msg) { super(msg); this.name = "BrowserDisconnectedError"; }
}

const mk = () => new MockLanguageModelV4({
  doStream: async () => ({
    stream: simulateReadableStream({ chunks: [
      { type: "stream-start", warnings: [] },
      { type: "response-metadata", id: "id-0", modelId: "mock", timestamp: new Date(0) },
      { type: "tool-input-start", id: "call-1", toolName: "enter" },
      { type: "tool-input-delta", id: "call-1", delta: '{"ref":"e1"}' },
      { type: "tool-input-end", id: "call-1" },
      { type: "tool-call", toolCallId: "call-1", toolName: "enter", input: '{"ref":"e1"}' },
      { type: "finish", finishReason: "tool-calls",
        usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 } },
    ]}),
  }),
});

const run = async (label, execute) => {
  const errors = [], toolEnds = [];
  const r = streamText({
    model: mk(), prompt: "go",
    tools: { enter: tool({ inputSchema: z.object({ ref: z.string() }), execute }) },
    onError: (e) => errors.push(e),
    onToolExecutionEnd: (e) => toolEnds.push(e),
  });
  const [toolResults, content] = await Promise.all([r.toolResults, r.content]);
  const te = content.find((p) => p.type === "tool-error");
  console.log(`${label}: toolResults=${toolResults.length} content=[${content.map(p => p.type)}]`);
  if (te) console.log(`  instanceof BDE=${te.error instanceof BrowserDisconnectedError}`);
  console.log(`  onError fired=${errors.length} onToolExecutionEnd=${toolEnds.map(e => e.toolOutput?.type)}`);
};

await run("succeeds", async () => "ok");
await run("throws  ", async () => { throw new BrowserDisconnectedError("brob"); });