Name the First Value model step's failures (#745)

[nick-neely]

Closes #745. The alert, retry, and recovery policy decision moved to #747.

Summary

The gateway SDK reports the probe's own 15-second deadline as a retryable 500, and the retry wait then throws AbortError: Delay was aborted. So both 2026-10-05 alerts show only that the deadline ran out; the upstream error was thrown away. The model step now runs its own attempts under a deadline it owns, and logs what each one did.

 model step (First Value path)
-  generateText(abortSignal: timeout(15s))          # SDK: up to 3 calls, hidden
-  on throw: log { step, reason: error.name }       # always "AbortError"
+  deadline = 15s, owned by the step
+  for delay in [0, 2s, 4s]                         # SDK default, now explicit
+    wait(delay) unless deadline fires
+    generateText(maxRetries: 0, abortSignal: deadline), cut off at deadline
+    deadline fired?  attempt = deadline
+    else             attempt = { status, type, retryable, generationId }
+    stop unless retryable
+  log first_value_check.passed | failed
+    { step, reason?, elapsedMs, attempts[] }       # allowlisted tokens only

Alerting is unchanged: any failed cheap step still opens first_value_path on that pass. The one policy difference is that a gateway Retry-After is no longer honoured. The SDK would have waited up to 60s for it, past the deadline anyway.

Evidence

• Before: with the pinned SDK and a fake gateway, a hung call and a slow 503 both reject as AbortError, so the log says { step: "model", reason: "AbortError" }.
  After: first-value-check.test.ts (fake timers, no paid calls):

  hung call               -> reason deadline, 1 attempt { deadline, 15000ms }
  503 at 14s, then wait   -> reason deadline, attempt keeps status 503 / type / generationId
  401 permanent           -> 1 attempt, no backoff
  503 then pass           -> passed, failed attempt still logged, 2800ms
  503 x3                  -> 3 calls, 6000ms
  call ignores the signal -> ends at 15000ms as deadline
  hostile error fields    -> message, body, untokenlike name/type/id dropped
  real generateText path  -> 3 model calls total (9 if maxRetries: 0 is removed; checked)
  

• Local: pnpm verify typecheck, lint, and tests passed (344 test files). The web next build hung once at "Creating an optimized production build" and passed on retry (exit 0). pnpm coverage:ci && FALLOW_AUDIT_BASE=origin/main pnpm fallow:ci passed with no findings.

Merge Danger

Door: two-way

Reverting restores the old probe. No schema, config, or data changes.

Blast Radius: probe-only

Only the synthetic First Value model step changes. A failing model step can now make up to three calls (it already could through the SDK default), and each pass adds one first_value_check.passed info log line. grounded_eve_answer, the daily claim, metering through the model-call entry point, pinned provider, and zero-retention routing are untouched.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Improved reliability of the First Value check with up to three model-call attempts for retryable failures, bounded by a 15-second deadline. Each attempt reports its outcome and elapsed time.
  • Improved failure reporting when the model is unavailable or a call times out.

• Documentation

  • Clarified how retries affect Usage Ledger entries and how to verify model-call outcomes in operational logs.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
tendnote-web	Ready	Preview	Oct 6, 2026 6:39am UTC

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: c50ef6ad-96db-4345-9974-56cd69b7905e

📥 Commits

Reviewing files that changed from the base of the PR and between 8f0300d and d66c719.

📒 Files selected for processing (4)

• apps/web/src/lib/first-value-check.test.ts
• apps/web/src/lib/first-value-check.ts
• docs/adr/0263-reliability-indicators-are-read-by-the-recovery-cron-and-eve-is-asked-daily.md
• docs/operations/synthetic-first-value-check.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

Walkthrough

The First Value model check now retries eligible gateway failures up to twice within a 15-second deadline. It records elapsed time and per-attempt outcomes, and limits logged error details to allowlisted fields. The tests and operational documentation cover retries, deadline outcomes, and the resulting logs.

Changes

First Value model check

Layer / File(s)	Summary
Model call and diagnostic contract apps/web/src/lib/first-value-check.ts, apps/web/src/lib/first-value-check.test.ts	The injected model call accepts an abort signal, and the default call forwards it with SDK retries disabled. Attempt records include elapsed time and allowlisted error fields. Tests verify the call setup and diagnostic fields.
Deadline, retries, and attempt logging apps/web/src/lib/first-value-check.ts, apps/web/src/lib/first-value-check.test.ts, docs/adr/0263-reliability-indicators-are-read-by-the-recovery-cron-and-eve-is-asked-daily.md, docs/operations/synthetic-first-value-check.md	The model step allows up to three calls within 15 seconds and waits two and then four seconds before eligible retries. Tests cover passing calls, retry limits, non-retryable errors, deadline handling, and safe log fields. The ADR and runbook describe retries, Usage Ledger calls, and verification logs.
First Value check integration apps/web/src/lib/first-value-check.ts, apps/web/src/lib/first-value-check.test.ts	The First Value check calls the model step after earlier checks pass. Tests verify that failed sign-in or admission prevents the call, and that a missing model ID produces a failure log with no attempts.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant FirstValueCheck
  participant modelStep
  participant callModel
  participant hostedModel
  participant logger
  FirstValueCheck->>modelStep: Run model check
  loop Up to three attempts within the deadline
    modelStep->>callModel: Call with abort signal
    callModel->>hostedModel: Generate text with SDK retries disabled
    hostedModel-->>callModel: Return result or gateway error
    callModel-->>modelStep: Return call outcome
    opt Retryable failure and time remains
      modelStep->>modelStep: Wait 2 or 4 seconds
    end
  end
  modelStep->>logger: Record elapsed time and attempt outcomes
  modelStep-->>FirstValueCheck: Return pass or failure

Loading

Merge Risk: ⚪ Minimal · up to d66c7

The First Value model check now retries transient gateway failures within a 15-second deadline and logs bounded diagnostics. The supplied evidence shows no concrete merge-blocking risk.

🚥 Pre-merge checks | ✅ 3 | ❓ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Linked Issues check	❓ Inconclusive	For #745, the model step now owns a 15-second deadline, makes at most three metered calls, records allowlisted attempt details, and preserves failed-attempt details when a retry passes. The added test…	Evidence from the existing alert/recovery test suites and the current operational guidance is needed to decide whether #745's alert-sequence test and policy-documentation criteria are met.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the First Value model step and its failure diagnostics, which are the main change.
Description check	✅ Passed	The description explains what changed, why it was needed, how retries and diagnostics work, related issue #745, impact, and reported testing. It does not use all template headings or select a Type of …
Out of Scope Changes check	✅ Passed	The code, tests, ADR, and runbook changes all support #745's First Value diagnostics, bounded retry, or verification objectives. I found no unrelated changes.

Full details: Linked Issues check

Explanation

For #745, the model step now owns a 15-second deadline, makes at most three metered calls, records allowlisted attempt details, and preserves failed-attempt details when a retry passes. The added tests cover retry limits, deadlines, permanent errors, and safe logging. The changed ADR documents the retry cap and deadline. I cannot establish from the reviewed evidence whether existing alert tests cover the required fail/recovery, intermittent-failure, missing/off-reading, and concurrent-pass cases, or whether existing operational guidance states the worst-case detection delay and recovery rule.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{I’m a rabbit counting calls,
Two and four seconds as time falls.
A deadline marks the final run,
Safe fields show what has been done.
I nibble logs and hop away.}

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — The PR changes an existing production recovery probe by introducing custom deadline, backoff, cancellation, and structured error handling, with up to three metered gateway attempts. Its scope is limited to the synthetic check and is well tested, but the new runtime policy and metering behavior are substantial enough to warrant human review.

Not approved because:

• Credit balance exhausted. Approvability relies on correctness review in order to determine eligibility

^{Review your spending limits in Billing settings. You can add or adjust custom eligibility rules. Learn more.}