[audit] Capture pi's tool events and hermes's working directory - #639
Open
SiddarthAA wants to merge 2 commits into
Open
[audit] Capture pi's tool events and hermes's working directory#639SiddarthAA wants to merge 2 commits into
SiddarthAA wants to merge 2 commits into
Conversation
Two adapters were discarding data their agents do emit. Both were found by
installing the CLI and driving a real session against a live provider, then
comparing what landed on disk to what the parser produced.
**pi dropped every tool event.** `lib/pi-sessions.ts` handled only `text` and
`thinking` content blocks; `toolCall` blocks fell through to the generic
"system" branch and the separate `role: "toolResult"` records were never
attached to anything. The file's own header explained this as "tool-call
blocks are not yet observed", and an unused `formatTimestamp` import was kept
alive with a `void` for "once Pi emits it" — so the gap was known, but the
premise behind it was wrong rather than merely stale.
Verified against pi 0.73.1 and 0.83.0: an assistant turn carries
`{type:"toolCall", id, name, arguments}` with `stopReason:"toolUse"`, and each
result arrives as its own record with a third role, carrying `toolCallId`,
`toolName`, `content[]` and `isError`. Results now attach to their call by id
rather than by position — pi emits them in call order today, but pairing by
order would break silently the first time it does not. pi records no duration,
so it is derived from the call/result gap, the same way the OpenClaw parser
does it. An orphan result (call not in this file) is still preserved as a
system entry rather than dropped.
**hermes contributed nothing to any cwd-scoped audit.** The adapter opened with
`if (opts.projects?.length) return []`, on the premise that Hermes sessions are
gateway sessions and therefore have no working directory. Verified against
hermes-agent 0.19.0: `sessions` carries real `cwd`, `git_branch` and
`git_repo_root` columns, and every `source='cli'` session populates them — so
`failproofai audit --project <repo>` silently reported zero Hermes findings for
a repo the user had actually driven Hermes in.
Both shapes are real, so both are handled: a session with a cwd now filters and
groups by working directory like Claude/Goose/Devin, while a Slack/Telegram
session — which genuinely is not in a repo — keeps its (profile, source) bucket
and is correctly excluded from a cwd filter. The data was already there;
`HermesSessionRef.cwd` was populated and the SQL already selected `s.cwd`.
Also corrects the goose adapter's docstring, which cited Hermes as the
cwd-less counterexample.
Tests build a real pi transcript and a real Hermes SQLite DB with both session
shapes. Nine of the new assertions fail against the previous code; the rest are
regression guards on the behaviour that was already correct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughPi session parsing now captures tool calls and pairs results by identifier. Hermes audit discovery now filters and groups sessions by working directory, with fallback grouping for cwd-less gateway sessions. Tests cover both behaviors. ChangesAudit and transcript handling
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
__tests__/audit/hermes-adapter-cwd.test.tsESLint skipped: missing config or dependency (missing-dependency). The ESLint configuration references a package that is not available in the sandbox. __tests__/lib/pi-sessions.test.tsESLint skipped: the ESLint configuration for this file references a package that is not available in the sandbox. lib/pi-sessions.tsESLint skipped: the ESLint configuration for this file references a package that is not available in the sandbox.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/pi-sessions.ts`:
- Around line 297-323: Extend the shared ToolResultInfo type with an error
field, then update the toolResult handling in the role-processing flow to
propagate raw.message.isError into block.result. Preserve the existing result
metadata and content while ensuring successful and failed Pi tool calls retain
their distinct error state.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c306044c-d3e3-42fe-9372-1acce425ca2f
📒 Files selected for processing (6)
CHANGELOG.md__tests__/audit/hermes-adapter-cwd.test.ts__tests__/lib/pi-sessions.test.tslib/pi-sessions.tssrc/audit/cli-adapters/goose.tssrc/audit/cli-adapters/hermes.ts
Comment on lines
+297
to
+323
| // Pi's third role: a tool result, on its own record, pairing back to an | ||
| // assistant turn's toolCall by id. Attaching it to that block is what | ||
| // makes the tool's OUTPUT visible — without this the call renders with | ||
| // no result and the audit path sees no `toolResultText` at all. | ||
| if (role === "toolResult") { | ||
| const callId = raw.message.toolCallId; | ||
| const block = typeof callId === "string" ? toolUseById.get(callId) : undefined; | ||
| if (block) { | ||
| // Pi records no duration on the result, so derive it from the gap | ||
| // between the call and its result. `startMs` is always present for | ||
| // a block we indexed; the fallback keeps the arithmetic total. | ||
| const startMs = (typeof callId === "string" && toolUseStartMs.get(callId)) || date.getTime(); | ||
| const durationMs = Math.max(0, date.getTime() - startMs); | ||
| block.result = { | ||
| timestamp, | ||
| timestampFormatted: formatTimestamp(date), | ||
| content: extractMessageText(content), | ||
| durationMs, | ||
| durationFormatted: formatDuration(durationMs), | ||
| }; | ||
| continue; | ||
| } | ||
| // Orphan result — the call was never seen (truncated file, or a | ||
| // resumed session whose earlier half is in another file). Fall | ||
| // through so the record is preserved rather than dropped. | ||
| } | ||
|
|
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check ToolResultInfo shape and whether isError/error fields exist and are consumed elsewhere.
ast-grep run --pattern 'interface ToolResultInfo {
$$$
}' --lang typescript lib/log-entries.ts
rg -n 'isError' lib/ src/ --type=ts -C3Repository: FailproofAI/failproofai
Length of output: 5295
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== log-entries interfaces around ToolResultInfo =="
sed -n '1,80p' lib/log-entries.ts
echo
echo "== pi-sessions record schema and toolResult construction =="
sed -n '1,170p' lib/pi-sessions.ts | sed -n '133,142p'
sed -n '260,330p' lib/pi-sessions.ts
echo
echo "== usages of ToolResultInfo and result.isError across codebase =="
rg -n '\bresult(?:\.isError|\[["'\'']isError["'\'']\])\b|\bisError\b|ToolResultInfo' . --glob '!node_modules' --glob '!dist' --glob '!build' -C 2Repository: FailproofAI/failproofai
Length of output: 21747
🌐 Web query:
Pi CLI Extension tool_result isError field PostToolUse result isError
💡 Result:
In the Pi CLI ecosystem, the isError field is a boolean property used within tool execution results to indicate whether a tool call failed [1][2][3][4]. Key details regarding isError and its usage in Pi extensions: 1. Core Functionality: The isError field is present in the ToolResult record [1][2]. When a tool execution concludes, the Pi harness determines the value of this field [3][4]. If set to true, it signals to the system and the model that the tool call did not succeed, which typically results in the emission of a failure event or state (such as PostToolUseFailure) [5][4][6]. 2. Tool Execution Logic: By default, many Pi tools (e.g., the bash executor) treat any non-zero exit code as an error, automatically setting isError to true [7]. This has been noted as a potential source of false positives for commands like grep or diff, which use non-zero exit codes to signal results (e.g., "no match") rather than actual system failures [7]. 3. Extension Hook (PostToolUse): Pi allows extensions to intercept tool results via the tool_result hook (sometimes referred to as afterToolCall in different API surfaces) [8][6][7]. This hook fires after a tool finishes but before the result is fully processed by the model [8]. - Extensions can access the incoming isError status to perform branching logic (e.g., only reacting to failed tools) [5][6]. - Extensions may be able to override the isError status by returning an updated object, allowing them to correct false-positive errors [6][7]. 4. Event Stream and Debugging: When running in modes like --mode json, the isError field is included in the structured event output, specifically within tool_execution_end or similar tool result events [3][4][7]. This allows external consumers to programmatically parse and respond to tool success or failure states [9][3].
Citations:
- 1: https://github.com/marcfargas/pi-test-harness
- 2: https://github.com/gaodes/pi-test-harness/blob/main/README.md
- 3: https://takopi.dev/reference/runners/pi/takopi-events/
- 4: https://github.com/terryso/open-agent-sdk-swift/blob/af6b1536/Sources/OpenAgentSDK/Core/ToolExecutor.swift
- 5: https://github.com/rullerzhou-afk/clawd-on-desk/blob/main/docs/guides/state-mapping.md
- 6: https://commandcode.ai/docs/mods
- 7: coding-agent: bash tool records grep/diff exit 1 ("no match") as
isError: trueearendil-works/pi#3051 - 8: https://github.com/giuliastro/HarnessTrim/blob/main/packages/adapter-pi/README.md
- 9: https://pi.dev/packages/pi-agent-browser-native
Add the error field to ToolResultInfo and attach isError from Pi results.
raw.message.isError exists for role: "toolResult" records, but ToolResultInfo only defines timestamp, timestampFormatted, content, images, durationMs, and durationFormatted. Add an error field to the shared result type and propagate the Pi flag into block.result; otherwise failed tool calls keep the same result shape as successful calls in the audit path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/pi-sessions.ts` around lines 297 - 323, Extend the shared ToolResultInfo
type with an error field, then update the toolResult handling in the
role-processing flow to propagate raw.message.isError into block.result.
Preserve the existing result metadata and content while ensuring successful and
failed Pi tool calls retain their distinct error state.
SiddarthAA
added a commit
that referenced
this pull request
Aug 5, 2026
Two audit adapters were discarding data their agents do emit — pi contributed zero tool events (only `text`/`thinking` blocks were handled, so `toolCall` blocks fell to the generic system branch and `role: "toolResult"` records attached to nothing), and the hermes adapter returned nothing at all for `audit --project <cwd>` on the premise that gateway sessions have no working directory, which its own `sessions` table contradicts. One conflict, in CHANGELOG.md, and it was structural rather than semantic: the PR branched when the top section was `## 0.0.16-beta.0 — 2026-07-31` and adds a single line beneath it. That heading no longer exists — the changelog was renumbered to 1.0.0-beta.x — so git could not place the line and marked the whole top of the file conflicted. Resolved by re-placing that one line under the current `## 1.0.0-beta.5` → `### Fixes`, verified absent beforehand so the merge cannot duplicate it. Nothing else conflicted. No branch here has ever touched `lib/pi-sessions.ts`, `src/audit/cli-adapters/goose.ts` or `src/audit/cli-adapters/hermes.ts`, and main has not moved them since the PR's base, so the five code files merged clean. tsc clean; 3095 unit tests pass, including the 19 the PR adds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F9eufSorsg96cwpfSdamAk
NiveditJain
added a commit
that referenced
this pull request
Aug 7, 2026
…632) * [failproofaid] Add empty Rust workspace and gated CI plumbing Stage 1 of the failproofai/failproofaid split: an empty Cargo workspace plus a rust-quality CI job gated on crates/*/Cargo.toml existing, so it goes green before any daemon code lands. Also fixes the bun cache key (hashFiles('bun.lockb') has silently never matched anything since this repo tracks bun.lock, not bun.lockb) and extends the version-consistency check to cover the new Cargo workspace version. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * [failproofaid] Add the daemon skeleton and IPC wire protocol Stage 2: crates/fpai-ipc (length-prefixed JSON framing, the ping/hook envelope, SO_PEERCRED/getpeereid peer verification) and crates/failproofaid (a Unix-socket server binding ~/.failproofai/run/failproofaid.sock at 0600 inside a 0700 dir, a flock-based singleton guard, and graceful SIGTERM shutdown). Hook requests get a stub "not implemented" error for now -- wiring them to a real warm Node/Bun worker is Stage 3. 28 Rust tests (unit + black-box binary integration tests spawning the real compiled binary), plus manual end-to-end verification against an independent Python client exercising ping/pong, the stub hook response, protocol-version mismatch, malformed frames, and an oversized length prefix. Testing caught two real bugs before they shipped: serde's rename_all on an enum only renames variant tags, not struct-variant field names (protocolVersion was silently serializing as protocol_version), and ensure_run_dir was unconditionally chmod-ing whatever directory FAILPROOFAI_DAEMON_SOCKET's parent resolved to -- now it refuses to touch a pre-existing directory it didn't create itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * [failproofaid] Wire the warm worker, TS daemon-client, and fail-closed path Stage 3: the full daemon-aware hook path, end to end. TypeScript side: - handler.ts: extract the core evaluation logic into evaluateHookEvent(), which takes the stdin payload as a parameter and returns {exitCode,stdout,stderr} instead of touching process.stdin/stdout directly -- callable repeatedly inside a long-lived worker process. handleHookEvent() keeps its exact existing signature/behavior as a thin wrapper, so the 1300+ line existing test suite needed zero changes. Adds a forceDecision option that registers a single synthetic policy and runs it through the real, unmodified evaluatePolicies() -- the fail-closed path gets correct per-CLI response shaping (Cursor's flat continue:false, Factory's exit-2, ...) for free, with no duplicated shaping logic. Also closes the process.cwd() hazard: a fallbackCwd option lets a warm worker use the *originating* CLI's cwd instead of its own fixed one when a payload omits cwd. - worker-server.ts + bin/failproofai-worker.mjs: the Node/Bun process failproofaid spawns. Listens on its own Unix socket (not stdio -- a stray console.log from a user's custom policy must never desync a shared framed channel), processing requests strictly sequentially so the globalThis-backed policy registry stays correct with zero changes. - daemon-client.ts: the thin client, real net.Server-tested framing matching crates/PROTOCOL.md exactly. ~150ms connect+roundtrip budget, null on any failure (no partial trust). isDaemonConfigured() gates the whole path on a new HooksConfig.daemonConfigured marker (global scope only) that nothing sets until Stage 4 -- inert on every machine today. - bin/failproofai.mjs: daemon-attempt-then-fail-closed wiring, byte-for- byte unchanged when not daemon-configured. - builtin-policies.ts: fixes the git-branch cache for a warm process -- it previously never actually cached anything (one-shot process), and reusing it unconditionally across many calls would silently serve a stale branch after a checkout. Now gated on .git/HEAD's mtime. Rust side: - worker.rs: spawns and supervises the worker, relays Hook requests to it, translating between the worker-facing protocol (no protocolVersion -- this process always spawns a version-matched worker) and the client-facing one (which does). - server.rs: Hook requests now relay through the worker instead of returning a stub error; any worker failure becomes a client Error response, never a hang. A live end-to-end test (real failproofaid, real bun-spawned worker, real block-sudo policy) surfaced a genuine process-leak bug during development: `sh -c "bun ..."` isn't guaranteed to exec(2) in place, so killing only the tracked PID could leave a live grandchild worker process orphaned -- reproduced live as three orphaned failproofai-worker.mjs processes still running minutes later. Fixed with process groups (spawn in a new group, kill the whole group) and piped rather than inherited stdio (an inherited fd on a worker that outlives its intended lifetime keeps a wrapping shell's pipe from ever seeing EOF). Test infrastructure also gained an RAII guard so a failing assertion cleans up its spawned worker exactly as reliably as a passing one. 29 Rust tests, all passing worker-server.ts/daemon-client.ts tests against real sockets (not mocks), full existing TS suite green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: add CHANGELOG entry for the failproofaid split (#632) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(ci): install bun in rust-quality before cargo test The live end-to-end test in crates/failproofaid/src/server.rs spawns the real TS worker via `bun bin/failproofai-worker.mjs`, but the rust-quality job only ever set up the Rust toolchain -- bun was never on PATH there. Failed on real CI with exit status 127 ("worker process exited before creating its socket") even though it passed locally, where bun happens to already be installed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * [failproofaid] Install failproofaid as a real OS service from failproofai config Stage 4: no separate `failproofai daemon install` command (explicitly rejected during planning) -- `daemon-service.ts` is a small install/ uninstall/status engine that `configure-wizard.ts` calls directly, the same relationship `config` already has with manager.ts's installHooks(). - daemon-service.ts: writes + enables a systemd --user unit on Linux (~/.config/systemd/user/failproofaid.service) or a launchd LaunchAgent on macOS (~/Library/LaunchAgents/ai.failproof.failproofaid.plist). resolveFailproofaidBinaryPath() points ExecStart/ProgramArguments at the real compiled binary directly -- never the eventual JS bin shim, which only exists for a user invoking `failproofaid` by hand, not for what a service manager should supervise. Resolves via (in order) an explicit test/dev override, the future @failproofai/failproofaid-<os>- <arch> npm package, or a locally-built target/{release,debug}/ failproofaid -- so this already works against this session's own cargo-built binary before Stage 5's packaging exists. - configure-wizard.ts: when the global ("Everywhere I code") scope is chosen on a supported platform, the wizard installs/starts the service and writes the daemonConfigured marker unconditionally -- no separate toggle, matching the product decision that this isn't an opt-in extra. A failed install never fails the wizard: the rest of setup already applied, and the machine simply stays on the in-process path since the marker is only set on success. The review screen lists the exact service file that's about to be written, alongside every other file the wizard already shows. Verified against a REAL systemd --user session (this sandbox has one) -- install, confirm `running`, uninstall, confirm fully removed, with the test backing up and restoring any real pre-existing unit rather than assuming a clean slate. macOS/launchd is unit-tested (plist content generation, path resolution) but not live-verified -- no macOS available here. Also closes a real safety gap caught while adding these tests: configure-wizard.test.ts already drives the wizard through scope "user" in several existing cases, and without mocking daemon-service.ts those tests would have shelled out to the real systemctl on whatever machine runs them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * [failproofaid] npm packaging, cross-compile CI, and warm-path fixes found via Docker verification Adds per-platform failproofaid binary packages distributed via optionalDependencies (packages/failproofaid-{linux,darwin}-{x64,arm64}), a failproofaid-shim.mjs bin entry that resolves and execs the right one, and a build-daemon.yml cross-compile matrix (4 real targets, staged into packages/*/bin/ for release). Bumps to 1.0.0-beta.0 across every version-carrying file, per explicit instruction — this is the daemon split, a major behavioral change on supported platforms. Two real bugs surfaced by driving a real Docker clean-install + a live daemon/worker relay end-to-end (not just unit tests): - The worker was spawned lazily on the first real request, taking ~700ms to cold-start — well past daemon-client.ts's 150ms fail-closed budget, so the very first hook call after every daemon (re)start failed closed even though the daemon was healthy. Fixed by pre-warming the worker in a background thread right after the daemon binds its socket (worker.rs, main.rs). - Every deny/instruct decision unconditionally awaited a live PostHog network POST before returning, regardless of opts.awaitTelemetryFlush — the warm worker passes that flag specifically to avoid this, but the inline telemetry call at handler.ts's "decisions that affect Claude's behavior" block never checked it. This made every real policy block through the daemon pay a live network round-trip (or up to 5s when PostHog is unreachable), which is precisely the enforcement path that most needs to be fast and reliable. Fixed to respect the same opt-out flushHookTelemetry() already honors, with a regression test. Also resolves the version-bumped Rust binary path (daemon-service.ts's resolveWorkerCommand()/Environment= threading) once more against the new version to confirm the fix still holds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: update CLAUDE.md for the failproofaid daemon split Corrects the CI job table (8 jobs across ci.yml, not the stale 4-job list, plus the separate path-filtered build-daemon.yml cross-compile matrix), adds the new Rust crates/daemon files to the project-structure cheatsheet, and records the deliberate decision to keep this repo's own dogfood hook configs on the in-process path rather than daemon-configured. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * [failproofaid] Fix 11 review findings on the daemon split, four enforcement-breaking CodeRabbit's review of #632 surfaced four bugs that each silently defeat enforcement, plus a set of smaller ones. Fixed with regression tests that fail against the old code. **macOS was broken outright.** `run_until` puts the listener in non-blocking mode. Linux discards that on accept (`accept4`); BSD-derived kernels inherit it. So on macOS `read_message` returned `WouldBlock` before the client's bytes landed, `handle_connection` read that as a malformed frame and answered with silence, and every hook call on macOS — a platform this PR ships launchd support for — fell through to the client's fail-closed deny. Accepted streams are now explicitly set blocking. **Worker restart never worked.** A Unix socket file outlives the process that bound it, so `socket_path.exists()` saw the *dead* worker's leftover file the instant a new one spawned, broke out of the wait loop, and handed `call()` a socket nothing was listening on — ECONNREFUSED on every request until the daemon restarted, which is precisely the crash-recovery path the loop exists to provide. Readiness is now a real `connect()`, the stale path is cleared before spawn, and `Drop` cleans up after itself. Both new tests fail on the old code with ECONNREFUSED. **`daemonConfigured` tracked the service manager, not a daemon.** It was granted the moment `systemctl enable --now` / `launchctl load` exited 0 — which a daemon that dies at startup also does — and never revoked on uninstall. Since `bin/failproofai.mjs` fails closed on that flag, either end of the lifecycle left the machine denying every hook event across all 11 CLIs, recoverable only by hand-editing `~/.failproofai/policies-config.json`. Install now waits for the service to reach *and hold* a running state (a `Type=simple` unit reports active the moment it forks, so a single reading waves through exactly the crash-at-startup case), and `uninstallDaemonService` clears the marker first and unconditionally. The marker write moves to `daemon-service.ts` as `setDaemonConfigured` so both ends share one implementation. **A 150ms budget covered policy evaluation.** On a daemon-configured machine a client timeout is a DENY, not a fallback — and that one budget had to cover the whole roundtrip, which `handler.ts` allows 10s per custom policy and `worker-server.ts` serializes. A slow-but-correct verdict produced the same block as a dead daemon, so users would see intermittent denials of legitimate tool calls. Split into a 150ms *connect* probe (a dead daemon still fails fast, adding no latency) and a 30s *response* budget matching worker.rs's own read timeout. Also: - `process.exit()` in the `--hook` path discarded unflushed stdout. Under every agent CLI that stdout is a pipe, so writes are async and exit drops what's buffered — measured: 2 MB written, 146 KB delivered. That truncates the decision payload the CLI parses, and on the fail-closed path drops the deny reason entirely. Both hook paths now drain first. - The worker's piped stdout/stderr were never read. The worker runs real policy code for the daemon's whole life, so a chatty custom policy eventually fills the pipe buffer and blocks it mid-write — every later hook call fails closed. Both pipes now drain on background threads into the daemon's own stderr, where systemd/launchd already capture it. - `worker-server.ts` decoded one frame per `data` event, stranding the second of two coalesced requests until a third write arrived. - Connections had no read/write deadline and no cap: a peer that connected and sent nothing held a thread for the daemon's lifetime. Now a 10s deadline plus a 64-connection ceiling. - Every `systemctl`/`launchctl` call was unbounded, so a wedged user session hung the wizard silently after the user pressed apply. - Daemon-install telemetry sent `err.message` verbatim — for writeFileSync/execFileSync failures that is an errno string carrying a `homedir()`-derived absolute path, i.e. the OS username. Only a bounded classification leaves the machine now; the full text stays in the local log. - The e2e harness piped the daemon's output without draining it (same buffer-fill hang) and orphaned a live daemon on any assertion panic. A `DaemonGuard` kills and reaps on drop, startup polls `try_wait()`, and failures report the daemon's own stderr. CI: - `darwin-x64` used the retired `macos-13` label. An unknown label doesn't fail — it never gets a runner, which is why that leg has been pending since the PR opened. Now `macos-15-intel`. - The release-artifact job shared a writable cargo cache between `pull_request` and `release` triggers, so a PR branch could seed an entry a later release run restores into a published binary. Restore-only on PRs, save on release/dispatch. - `persist-credentials: false` on the two checkouts that then compile third-party crates, so build scripts can't read GITHUB_TOKEN out of `.git/config`. Verified end to end against the real compiled daemon (deny, allow, worker-killed-mid-life restart, daemon-down fail-closed), plus `cargo test --workspace`, `bun run test:run`, `bun run test:e2e`, lint and tsc. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3D1CyJQDbxM25cNBgRrH7 * docs(changelog): fold the --locked release build into the CI entry Same subject as the surrounding sentence (hardening the job that produces the binary users install), so it belongs in that bullet rather than a new one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3D1CyJQDbxM25cNBgRrH7 * [failproofaid] Ship the daemon from GitHub Releases, not npm platform packages The four `@failproofai/failproofaid-<os>-<arch>` packages were the plan of record — declared as optionalDependencies, pinned to the root version, built by a 4-way cross-compile — but nothing ever published them. The workflow only uploaded the binaries as Actions artifacts and publish.yml was never touched at all, so all four names 404 on npm today and a released CLI would have resolved a daemon that does not exist. #634 fixes the pipeline either way; this removes the channel it would have had to publish, because the release assets have to exist regardless for anyone installing failproofaid on its own, and a second channel is a second thing to keep in step with the first. daemon-download.ts fetches `failproofaid-<os>-<arch>.gz` from the release tagged with this CLI's own version, verifies it against the published SHA256SUMS *before* decompressing, and installs it to `~/.failproofai/bin/failproofaid-<version>` by atomic rename, mode 0755. The URL is constructed from package.json's version rather than discovered through the API: no rate limit, no `releases/latest` redirect, and no way to run a daemon built from different source than the CLI talking to it. The versioned filename is what keeps an upgrade from overwriting a running binary (ETXTBSY) or silently repointing a live service unit. A bad checksum, a missing manifest entry and a failed fetch are refusals rather than warnings — what this writes is an executable a service manager runs at login. Only the install path downloads; resolveFailproofaidBinaryPath() stays a pure disk check, so the hook path can never block on the network. FAILPROOFAI_NO_DOWNLOAD=1 opts an air-gapped machine out while leaving an already-installed binary working, and FAILPROOFAI_DAEMON_BASE_URL points at an internal mirror (and at a local server in the tests). build-daemon.yml matches #634's copy so the rebase is a no-op: it gzips each binary and uploads that, which is also what makes the artifact's lost executable bit a non-issue. Verified: 13 new download tests against a real local HTTP server covering checksum mismatch, a manifest with no entry, a 404, the disabled-downloads opt-out and the atomic install; the daemon-service resolution tests now run against a scratch HOME so a developer machine with a real daemon cannot flake them; and a clean `npm pack` + container run confirms the shim reports the config hint with nothing installed and execs an overridden binary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbXKFnpRXBwsrvUJDdySky * [failproofaid] Static musl binaries and a system-scope service Two defects the 1.0.0-beta.0 release surfaced on real machines. The Linux binaries linked against the build runner's glibc, and `ubuntu-latest` is now 24.04 (glibc 2.39), so they refused to start on Ubuntu 22.04, Debian 12, RHEL 9 and Amazon Linux 2023 — measured against real containers, not predicted. Both Linux legs now target `*-unknown-linux-musl` and link statically, which has no glibc floor at all; an older runner would only move the floor (22.04 is 2.35, still above RHEL 9's 2.34). The build asserts staticness on the artifact itself, because a "static" build that came out dynamic still runs on the runner that made it and fails only on the users' distros. The service was a systemd --user unit, which only runs while its user manager does: without lingering that manager does not start at boot and stops with the last session. So the daemon died on logout — and on a daemon-configured machine an unreachable daemon fails closed, so any agent running without a login session (detached tmux, cron, a CI runner) then hit denials. It is now `/etc/systemd/system/failproofaid@<user>.service` with `User=<user>` and `WantedBy=multi-user.target`, enabled via `systemctl enable --now`; macOS moves from a LaunchAgent to a LaunchDaemon with `UserName`. Root-installed, never root-run: everything it touches still lives in one user's home and is peer-checked against that uid. The two costs of system scope are handled rather than assumed. Install needs root, so `canElevate()` probes `sudo -n` BEFORE writing anything and, failing that, returns the exact commands to run (classified as `needs_root`) instead of half-installing — never an interactive prompt, which would be unreadable under the wizard's TUI. And a system unit inherits no login environment, so `FAILPROOFAI_WORKER_CMD` now names an absolute runtime via `process.execPath`: a bare `node` resolves for the wizard and then fails inside the service on every nvm install, silently, which is the same class of bug CLAUDE.md documents for the dev hook. Any pre-existing user-scope daemon is stopped and removed on both install and uninstall. It holds the same singleton flock, so leaving one behind would make the new service lose the race and leave the machine fail-closed against a daemon that never came up. The unit is named per user so a second person on the same box cannot silently steal the first's service; every field in it is user-specific anyway. Status needs no privileges — `systemctl status failproofaid@<user>`, exposed as `daemonStatusCommand()`. No version bump here: `block-version-bumps` reserves that for a `luv-cut-*` branch, which is where 1.0.0-beta.1 gets cut. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbXKFnpRXBwsrvUJDdySky * test(daemon): stop the service tests reaching the real release CI caught what a local run could not. `installDaemonService()` downloads the daemon when nothing is resolvable, so the test asserting "fails cleanly when the binary cannot be resolved" actually fetched the real 1.0.0-beta.0 asset from GitHub Releases and installed it — the install then succeeded, failing that assertion, and the binary it left in the runner's home broke a later test that expects win32 to resolve nothing. It passed locally only because this sandbox has no network in the test environment. Downloads are now off for the whole file (the download path itself is covered in daemon-download.test.ts against a local HTTP server), and the two tests that assert "nothing is installed" run against a scratch HOME so a machine that really has a daemon — a CI runner that just ran the lifecycle tests, a developer laptop — cannot flake them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbXKFnpRXBwsrvUJDdySky * [failproofaid] Prompt for sudo in-process, and ask for the daemon first `sudo failproofai config` was the advice beta.1 printed when it could not elevate, and it is actively wrong. Under sudo `homedir()` is /root: the hooks land in root's settings, `daemonConfigured` is set for root, the binary downloads to /root/.failproofai/bin, and the unit is generated with `User=root` — the entire user-scope design undone silently, on the one path a user follows when something already went wrong. The wizard now refuses to run under sudo when SUDO_USER shows a real user behind it, and names the account to re-run as. A genuinely root-only environment, which has no SUDO_USER, still works. Service installation becomes step 0. It is the only step that needs a password, so asking there means `sudo -v` prompts on a clean terminal instead of firing from underneath a drawn TUI screen, where the prompt is invisible and the typed characters land in a redrawn frame. That single prompt caches the credential for the run, which is what keeps the install itself non-interactive — no sudo -n failure, no half-written unit. The daemon is no longer inferred from the scope either. It is machine-level — one service for every project on the box — so step 0 is where the user consents to it, and a project-scope setup can have one too. Declining, or failing to authenticate, never costs the rest of the setup: the wizard says so and applies everything else, exactly as a machine with no daemon behaved before. Six existing tests asserted the old scope-gated flow — the behaviour this changes — so they move with it, and three new ones pin what actually matters: that sudo is primed BEFORE any other question (ordering is the whole fix), that declining never touches sudo or the service, and that a refused password still applies the rest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbXKFnpRXBwsrvUJDdySky * [failproofaid] Close three review findings in the service install All three were introduced with the system-scope service, and all three are real. The staging file for the privileged write was named from pid + timestamp in the shared temp dir. That is guessable, and writeFileSync follows a symlink already sitting at the path — so on a multi-user box another local user could pre-create it and have `install` copy content they control into /etc/systemd/system as root. Staging now happens inside a mkdtempSync 0700 directory, whose name cannot be pre-created. FAILPROOFAI_WORKER_CMD joined two paths with a space and no quoting, and the daemon runs that value through `sh -c` (WorkerCommand::Shell in crates/failproofaid/src/worker.rs). Any path containing a space split into fragments and the worker never started — ordinary on macOS (/Users/First Last/…), and newly likely because the absolute process.execPath replaced a bare `node`. Both halves are shell-quoted now; systemd's Environment="…" quoting does not help, because that protects the unit parse, not the later shell split. The launchd label was still a fixed string while the systemd unit was already per-user — so a second Mac user's install overwrote the first's daemon (UserName, the ExecStart path under their own ~/.failproofai/bin, their log paths) and their uninstall deleted it. Label and plist path are namespaced per user, and the shared 1.0.0-beta.1 LaunchDaemon is stopped and removed on install like the legacy LaunchAgent, since it holds the same singleton flock. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbXKFnpRXBwsrvUJDdySky * feat: reconcile cloud-managed policy generations * feat(failproofaid): poll cloud-managed policy deployments * fix: let setup continue with no policy bundles selected The "What should we guard against?" step in `failproofai config` carried `minSelected: 1`, so anyone who wanted only their own custom policies — or who meant to pick bundles later — reached it and could not go forward. The only feedback was "Select at least 1", which explains the rule without offering a way past it, on the first screen a new user sees. Nothing downstream ever required a non-empty set. `installHooksImpl`'s explicit-array branch documents itself as "may be empty", `replace: true` makes that empty set the full enabled set at the scope, and `summarize([])` already renders "none". The wizard's own guard was the whole obstruction. So the minimum is gone and the hint says `· none is fine`, because a step that merely stops rejecting you still looks like one you are failing. Hooks install either way, so enforcement can be switched on later without running setup again — which is what makes an empty answer legitimate rather than a dead end. The review screen now reads "none enabled (add later: failproofai policies --install)" instead of "0 enabled": at zero, a bare count reads like the wizard dropped the selection rather than recorded it. The assistants step keeps its minimum, and a test pins the asymmetry. An empty CLI list there does not mean "no assistants" — `installHooksImpl` falls back to ["claude"] — so waving it through would silently install for a CLI nobody chose. The two steps look alike and must not behave alike. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: pause enforcement for one session, time-boxed A policy blocks legitimate work and the only exits were editing config or uninstalling. Config is the wrong instrument: it is persistent, merged across project/local/global, and routinely committed — a paused state written there outlives the session that asked for it and reaches everyone who checks out the branch. So `failproofai config --pause` writes session state, not config: `~/.failproofai/state/sessions/`, owner-only, atomic, keyed by a digest of the session id because twelve CLIs mint their own ids and nothing stops one containing a path separator. Disk is the source of truth rather than the daemon. Most machines have no daemon, and even where one runs, the CLI writing a pause is a different process from the hook reading it. A daemon may cache this; it must not own it. Expiry is evaluated at READ time against the clock, never by a sweeper, and there is no unbounded form: 30m default, 8h ceiling that config may lower and never raise. The failure mode worth engineering against is not "the pause didn't work", it is "the pause silently never ended" — so a file left by a crash is inert rather than a resurrected pause, and a corrupt file reads as NOT paused, failing toward enforcement. Scope is local only. Builtin, explicit-custom and convention policies are suspended; cloud-managed assignments keep firing, the same exemption `disabledCustomPolicies` already honours. A locally-issued command able to switch off a centrally assigned policy would make cloud enforcement decorative. Session resolution takes no argument because the activity log already records `sessionId` with `cwd` and a timestamp, so "newest session in this directory" is derivable from data we write anyway. With no recent match it refuses rather than guessing — pausing the wrong session leaves someone believing enforcement is off while it is on. Activity rows written under a pause carry `pausedBy`/`pauseExpiresAt`. Without them the log asserts a clean window over exactly the window nothing was enforced, which is worse than no log. `block-self-pause` (default on) stops an agent issuing it. A pause the agent can reach is not a guardrail — one shell-out suspends every other policy and survives the turn. It is not redundant with `block-failproofai-commands`: that anchors on a command boundary, so `npx -y failproofai config --pause` never matched it, and being broad it is plausibly switched off so agents can run `failproofai audit`. It stops the direct attempt, not the class; an alias or wrapper still reaches it, and closing that means the pause cannot originate from a tool call at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop the wizard tests writing into the repo's own config Three tests in configure-wizard.test.ts apply at PROJECT scope, and project scope resolves its config path from process.cwd() — which during a test run is this repository. So every run appended `"customPoliciesEnabled": false` to the tracked .failproofai/policies-config.json, and the next `git add -A` committed custom policies switched off for everyone who pulled. That is how it reached main. The file already isolated HOME, and that could never have caught this: project scope does not consult HOME at all, so the isolation and the defect were on different axes. Redirect the resolved path for the cwd-derived scopes into a temp dir rather than stubbing the write, so the real setCustomPoliciesEnabled still runs and stays under test — the leak proved it writes, and mocking it away would have removed the only coverage of that. User scope deliberately keeps the genuine HOME-derived path: the daemon tests read `daemonConfigured` back from it, and redirecting it too moved that file out from under them. Pinned by a test that reads the repo's own config before and after an applied project-scope run and asserts it is byte-identical. Verified to fail against the pre-fix code rather than assumed to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: show a paused machine in the dashboard The activity view had no idea a pause existed. Rows evaluated during one look exactly like rows where every policy ran and allowed, so the log asserted a clean window over precisely the window nothing was enforced — worse than no log, because it reads as evidence. Three pieces. A banner above the stats while any pause is live, a `paused` pill beside the decision badge so unenforced rows can be scanned for, and a note in the row detail saying an allow there proves nothing. The banner is fed by live pause state, polled independently of the activity table rather than derived from the rows on screen. A pause set seconds ago has produced no rows yet, and that is exactly when someone needs telling the machine is unguarded — so an absent banner has to mean "enforcing", which it cannot if it is inferred from history. It also re-filters by expiry on render and on a timer: a short pause can lapse between polls, and a banner outliving its pause claims an exposure that has ended. All three state that cloud-managed policies keep enforcing and how to end it early. Without the first the banner overstates how exposed the machine is; without the second the only visible exit is waiting. Extracted to app/components/pause-notices.tsx rather than left in the 1700-line client file, so they can be rendered in isolation under test. Doing that caught formatRemaining(30s) returning "1m": Math.round was rounding up, telling someone they had more time before enforcement returned than they did. A countdown to guardrails coming back must never round up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: enrol a machine with Failproof Cloud from the CLI Until now there was no supported way to connect a machine — the docs told operators to put FAILPROOFAI_CLOUD_TOKEN in the daemon's service environment, and nothing wrote it for them. That advice is unsafe. writePrivilegedFile installs /etc/systemd/system/failproofaid@<user>.service at mode 0644 — root-owned and world-readable — and the launchd plist likewise. An Environment= line there hands an organization-scoped bearer key to every local user, and `systemctl show` prints it back at no privilege at all. So the credential goes to ~/.failproofai/cloud.json at 0600, and the daemon reads it. Two consequences fall out of that which matter as much as the fix: enrolment, token rotation and disconnect touch no privileged path, so none of them need root; and enrolment stops being welded to service installation, so a daemon already running can be connected. The daemon re-resolves enrolment on EVERY poll, not once at startup. That is load-bearing, not tidiness: this is a system unit, restarting it needs root, and noticing a credential only on restart would put `sudo systemctl restart` back into the flow the file design exists to avoid. The same loop now serves enrolled and unenrolled machines, choosing its interval per tick so both documented knobs keep their meaning, and degrading to integrity-only — last known-good generation retained, tampering still repaired — when credentials are absent or broken. --connect verifies before it writes, making the exact request the daemon will make, and separates 401 (token rejected) from 403 (key lacks policies:pull) from unreachable. A credential that does not work is worse than none: --status would then report a connection this machine does not have. Plain http to a non-loopback host is refused since the token is a bearer credential; localhost stays allowed for the documented local walkthrough. The token is never printed. Env still wins over the file, so CI, containers and the existing tests are untouched. Verified end to end against the real daemon and a stand-in cloud: not enrolled and never contacting the server; enrolled while the daemon kept running and was never signalled; generation activated with byte-identical artifacts and an accepted bearer token; then --disconnect stopped polling with the last known-good generation left on disk. The run also caught the warning claiming failproofaid "is not installed" while one was plainly running — it reads the service manager — now worded "not installed as a service". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: attribute each decision to the policy that made it A cloud policy's revision existed only inside its display name ("cloud/org-guard@7/…"). So the one question centrally-managed policy has to answer — which rollout produced this decision — could be answered only by re-parsing our own label, and could not be filtered or aggregated at all. Activity rows now carry `policySource` (builtin/custom/convention/cloud), `cloudPolicyId` and `cloudRevision` for a cloud decider, and `cloudGeneration` on every row of a managed machine. Attribution is a lookup, not a parse: the map is built where each policy is registered, keyed by the exact name the evaluator reports back. A builtin is anything absent from that map, which makes the absence meaningful rather than missing, and means nothing has to agree about a prefix format twice. The generation is recorded even when a LOCAL policy decided. "What was deployed here" is a different question from "what decided", and only the former separates a rollout that changed no outcomes from one that never reached the machine. It is omitted rather than written as 0 on an unmanaged machine — a literal zero would read as a deployed generation. Rows written before this carry no policySource and are excluded from every source filter rather than guessed into a bucket from their name prefix. That inference would be easy and sometimes wrong, and when the whole point is proving which rollout decided something, a wrong attribution is worse than a missing one. The dashboard gains a `source` filter and shows both facts in the row detail. Adding the filter surfaced that the clear-filters button never reset it, which would have left a filter active while claiming to have cleared everything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: chart enforcement over time on the activity view The stats bar reports totals, which structurally cannot answer *when*. A deny spike at 14:00 and an all-day trickle produce identical numbers and mean very different things. Form is EMPHASIS — denies carry the accent, total volume recedes to context — rather than a categorical scale per policy source. That was not a style preference: the design system allows exactly two accent hues and there are four sources, so a four-colour categorical palette fails adjacent-pair separation by construction. Running the validator on the three status colours confirmed the shape of the problem before any of this was drawn. Deny keeps the red the decision badges already use, so one fact has one colour on the page. Empty buckets are emitted across the whole window instead of dropped. Omitting them compresses a quiet period into a line that reads as steady activity across an outage — a zero is data. Raw SVG, no charting dependency, matching the house style: crosshair and readout on hover, a legend stating both totals so identity is never colour alone, recessive axes, and an aria-label pointing at the activity table as the text equivalent. Bucketing is a pure function tested apart from the component. Writing those tests caught an event landing exactly on `now` being dropped by naive flooring — the newest event, and the one most worth seeing. The component is bounds-tested too, because the colour validator says nothing about layout and a coordinate outside the viewBox is clipped silently and reads as missing data. Not visually reviewed in a browser: the chart needs live activity data to appear, so the bounds test stands in for overflow but not for label collisions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: observe mode for cloud-managed policies An assignment now carries an effect. `observe` downloads, verifies and EVALUATES the policy exactly as any other, then discards the verdict and records what it would have been. Evaluating and discarding is the whole point: a policy that did not really run would measure nothing about the rollout being trialled, which is the only reason to trial it. A policy that throws or times out while observing is recorded as an ALLOW, because that is what it would have been in enforce mode. Recording it as a would-deny would overstate the policy's reach and argue for promoting something that does not work. The effect is carried into active.json, not just desired state — otherwise an observe-mode policy starts enforcing the moment the daemon restarts and re-reads its own manifest. Omitted means enforce at every layer: the opposite default would let a server predating observe mode silently downgrade a fleet to observation. An unrecognised effect is refused rather than guessed, since guessing means either enforcing what was meant to be watched or watching what was meant to be enforced. Separately, this removes deny_unknown_fields from the desired-state types. They parse a SERVER response, and daemons update on their own schedule, so strictness there meant the first field cloud ever added would make every older daemon fail to parse desired-state and silently stop pulling — a fleet stranded on whatever generation it happened to hold, with no error anyone would think to look for. Strictness stays on the manifests we author ourselves. The whole-chain test now sends an unknown field on every poll to keep it that way. Verified end to end against the real daemon and real hook path: an enforcing cloud policy denies, an observing one does not and is recorded with its id, revision and the decision it would have made, and a local pause suspends builtins while cloud policy keeps enforcing. 27/27. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(audit): capture pi's tool events and hermes's working directory Two adapters were discarding data their agents do emit. Both were found by installing the CLI and driving a real session against a live provider, then comparing what landed on disk to what the parser produced. **pi dropped every tool event.** `lib/pi-sessions.ts` handled only `text` and `thinking` content blocks; `toolCall` blocks fell through to the generic "system" branch and the separate `role: "toolResult"` records were never attached to anything. The file's own header explained this as "tool-call blocks are not yet observed", and an unused `formatTimestamp` import was kept alive with a `void` for "once Pi emits it" — so the gap was known, but the premise behind it was wrong rather than merely stale. Verified against pi 0.73.1 and 0.83.0: an assistant turn carries `{type:"toolCall", id, name, arguments}` with `stopReason:"toolUse"`, and each result arrives as its own record with a third role, carrying `toolCallId`, `toolName`, `content[]` and `isError`. Results now attach to their call by id rather than by position — pi emits them in call order today, but pairing by order would break silently the first time it does not. pi records no duration, so it is derived from the call/result gap, the same way the OpenClaw parser does it. An orphan result (call not in this file) is still preserved as a system entry rather than dropped. **hermes contributed nothing to any cwd-scoped audit.** The adapter opened with `if (opts.projects?.length) return []`, on the premise that Hermes sessions are gateway sessions and therefore have no working directory. Verified against hermes-agent 0.19.0: `sessions` carries real `cwd`, `git_branch` and `git_repo_root` columns, and every `source='cli'` session populates them — so `failproofai audit --project <repo>` silently reported zero Hermes findings for a repo the user had actually driven Hermes in. Both shapes are real, so both are handled: a session with a cwd now filters and groups by working directory like Claude/Goose/Devin, while a Slack/Telegram session — which genuinely is not in a repo — keeps its (profile, source) bucket and is correctly excluded from a cwd filter. The data was already there; `HermesSessionRef.cwd` was populated and the SQL already selected `s.cwd`. Also corrects the goose adapter's docstring, which cited Hermes as the cwd-less counterexample. Tests build a real pi transcript and a real Hermes SQLite DB with both session shapes. Nine of the new assertions fail against the previous code; the rest are regression guards on the behaviour that was already correct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(changelog): add the pi/hermes audit-adapter fixes (#639) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [collector] Add the fault-isolated collector host, inert until configured P0 of turning failproofaid into the single collector for session logs and hook activity. This lands only the part the whole single-process design rests on, and nothing that ships data yet. **Why a separate crate.** failproofaid gates every tool call on the machine — its CLI fails closed, so an unreachable daemon denies rather than falls back. That crate stays small and its tests stay fast. Collection is a much larger body of code with a far lower blast radius, and it should be buildable and testable without standing up a socket server, so it lives in `fpai-collect`. **Why the enforcement path is untouched.** The obvious move — make the whole daemon async — would mean rewriting `server.rs` (whose non-blocking/BSD-accept handling is the reason macOS works at all), `worker.rs`'s child supervision, and `fpai-ipc`'s sync `Read`/`Write` generics, then re-earning trust in the exact code path that gates tool calls. Not worth it to host a background uploader. The collector instead owns its own thread and its own Tokio runtime; the accept loop is byte-for-byte unchanged. **The three guarantees, each with a test that fails without it:** - Own thread, own runtime — and none of it when there is nothing to run. An empty task list starts no thread and no runtime, so a machine that has not opted in pays nothing for this code existing. - A panic is contained, counted and restarted with backoff, never propagated. Panics and errors are counted separately: a panic is a bug in a transform, an error is usually the environment, and conflating them would hide the former. A panicking task also does not disturb its siblings. - Shutdown is bounded. The collector observes the daemon's existing shutdown flag rather than adding a second signal path, backoff sleeps are interruptible so exit does not serve out a 60s wait, and `join_with_flush` abandons a wedged task at its budget rather than blocking process exit. `Shutdown` is handed to each task body, not just checked between attempts. The first version only checked between attempts and a test caught it immediately: every real source is a poll loop, so without this a task could only be stopped by expiring its flush budget — killed mid-iteration instead of exiting after persisting its cursor. Wired into main.rs behind an empty task list, so it is inert on every machine until the ingest configuration lands. Verified against a real daemon: a block-sudo deny still returns correctly through the socket, and SIGTERM still exits cleanly with the socket removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(changelog): add the collector host entry (#640) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Revert "feat: chart enforcement over time on the activity view" This reverts 98dec6d2. Policy observability belongs in the AgentEye dashboard, not the local one — a chart per machine answers "what did THIS laptop do", which is not the question an operator watching a fleet is asking. Nothing else depended on it: the attribution fields it read (policySource, cloudRevision, cloudGeneration) stay, and are what the AgentEye view will be built from once decisions actually reach cloud. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [collector] Add ingest config and the spool writer Continues P0. The daemon now resolves where to send events and what to collect, and can durably write a batch. Still ships nothing — no sources are wired — but the credential, spool layout and supervision they all depend on are in place. **The credential does not go in policies-config.json.** The plan said it would; checking the actual permissions said otherwise. `hooks-config.ts` writes that file with a bare `writeFileSync`, so it inherits the umask and lands at 0664, inside `~/.failproofai/` which is itself 0775. An API key there is readable by every local user on the box. So the credential lives alone in `~/.failproofai/ingest.json`, created at 0600 with the mode applied at open time rather than chmod-ed afterwards, and writing it tightens the home to 0700 — a 0600 file under a world-traversable directory is still reachable. `~/.agenteye/cli.json` already stores its session token at 0600, so the correct precedent existed. Everything non-secret (which streams are on, verbosity, redaction, environment) stays in policies-config.json under a `collector` block, where it is readable and diffable. **Two independent opt-ins.** A configured key does NOT start session collection: transcripts carry prompts, file contents and whatever was pasted into a terminal, so `sessions` defaults false and must be turned on deliberately. `hooks` defaults true — it carries decisions and tool names, never file contents. **Both spool directories are watched.** Derived batches go to `~/.failproofai/spool/`, but `~/.agenteye/events/` is watched too, so the Python SDK and any custom agent keep being collected with nothing to reconfigure when agenteye-collector retires. That is the point of superseding it rather than replacing it. **A config error disables collection loudly; it never stops the daemon.** The CLI fails closed, so a daemon that refused to boot over a malformed ingest.json would deny every tool call on the machine. Malformed JSON is still an error rather than being read as "absent" — quietly disabling collection over a stray comma is the silent failure this project exists to remove. An `environment` containing a comma is rejected outright, because ingest skips such lines server-side and every event from the machine would vanish with nothing on this end to show for it. The spool writer is a port with the incident specifics rewritten. Its three invariants each have a test: writes are atomic (tmp → fsync → rename, and `.tmp` is not `.jsonl` so a partial file is never visible); no written line can exceed the batch cap, since a line larger than one request could never be delivered and would be retried at the same size forever; and truncation is deterministic, because the server dedups on a content hash. 29 tests in the crate. Verified against the real daemon: no config starts nothing, a key logs the resolved endpoint and stream state, and a malformed file logs the exact syntax error while the socket keeps serving. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(changelog): add the ingest config and spool writer entry (#640) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [collector] Add the uploader, completing the delivery path Last piece of P0's plumbing. Batches written by the spool writer can now be delivered, retried, and — when they cannot be — parked without ever being lost. Still no sources, so nothing is produced to deliver yet. A port with the incident specifics rewritten. Four properties are load-bearing and each has a test, because each one's absence loses data rather than merely slowing delivery: **The timeout is per-read, not per-request.** A whole-request timeout also bounds streaming the body, and the body is re-sent in full on every retry — so a large batch on an ordinary uplink can never finish and burns its entire retry budget failing identically. `read_timeout` bounds progress instead of total size. On its own it bounds nothing (a server trickling one byte inside every window holds the request open forever), so a generous total cap stays as a backstop, set to a large multiple so it only catches a stuck request and never a merely slow one. **A 2xx is not automatically a success.** Ingest answers `{"accepted":N,"skipped":M}` and silently skips any line it will not store. Without reading that body, a batch the server discarded entirely is indistinguishable from a perfect upload — which is exactly the shape a systematically malformed transform takes. `accepted == 0 && skipped > 0` is logged at error and counted separately. **`failed/` is a retry queue, not a graveyard.** The filename carries the retry state (`<base>.a<N>[.c<STATUS>].jsonl[.poison]`) so a rename is the only atomicity needed; a sidecar could desynchronise from the batch it describes. A 4xx records its status and stops being auto-retried, because a rotated key or wrong URL fails identically until fixed and retrying burns the budget of batches that could succeed — except 408 and 429, the two that mean "try again". Poison files deliberately do not end in `.jsonl`, so every scan skips them for free. Nothing is ever deleted, and a name collision gets a numeric suffix rather than overwriting: a parked batch is the last copy of data the server does not have. **Oversized batches are split in memory.** Writing chunks to disk beside the original would create files the watcher had never seen, so it would post them concurrently with the upload in progress — the same payload delivered twice. Backoff jitter is derived from the clock rather than a PRNG, so the crate needs no `rand`: jitter has to spread retries after an outage, not be unpredictable. `rustls-tls` rather than native-tls, so the four cross-compiled targets gain no OpenSSL. Validated against the running AgentEye server, not only wiremock: a valid batch returns `{"accepted":2,"skipped":0}`, a batch missing `session_id` returns `{"accepted":0,"skipped":1}` (the fully-skipped case), and a bad key returns 401 — the three paths the code branches on. 87 workspace tests, clippy clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(changelog): add the uploader entry (#640) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [collector] Add the watcher and sweeper, completing P0 The native spool path is now live end to end: a batch published into either spool directory is delivered to the ingest endpoint. Verified against a running AgentEye instance — an SDK-shaped batch written to `~/.agenteye/events/` was picked up and is queryable server-side, with no CLI-specific code involved. **Two paths, different jobs.** The watcher exists for latency; the sweeper is what actually guarantees delivery. Filesystem events are lost in every way that matters — the daemon was not running, the watch failed to register, the queue overflowed, the filesystem reports nothing. A periodic scan has none of those failure modes. If the watcher were deleted tomorrow nothing would be lost, it would just arrive a sweep later. That asymmetry is why a failed watch registration is logged and shrugged off rather than being fatal, and why an unwatchable directory does not stop the task (on a filesystem without event support it would otherwise restart forever). **The watcher subscribes to renames, not just creates.** The spool publishes a batch by renaming it into place, which Linux reports as IN_MOVED_TO and macOS as a create. A watcher subscribed only to `Create` registers successfully, logs nothing, and delivers nothing on Linux — and because the sweeper covers a minute later, the bug reads as latency rather than breakage. There is a test for exactly this; removing the rename arm fails it with that diagnostic. **Both tasks share one Delivery.** One semaphore and one in-flight set, not one each. With separate sets a batch the watcher is mid-upload on is invisible to a concurrent sweep and both POST it; with separate semaphores the process issues twice the intended concurrency exactly when a backlog is draining. The in-flight claim is an RAII guard so an early return or a panic cannot leak it — a leaked claim makes that batch permanently invisible to both paths: undelivered, on disk, never retried. Concurrency is capped at 8 rather than the standalone collector's 64. This runs in the process that answers the enforcement socket, and 64 simultaneous TLS handshakes is a lot of CPU to put behind a hook call that must return in milliseconds. A backlog drains slightly slower; tool calls stay fast. Sweep order differs by directory and it is deliberate: the spool is newest first, so when a backlog cannot clear in one pass the events someone is looking at now arrive first; `failed/` is oldest first, because a parked batch is the last copy of data the server does not have and the one waiting longest is most at risk. Parked batches are retried on a far slower cadence so a backlog there cannot starve fresh events of upload permits, and anything poison or carrying a definitive client status is skipped — it fails identically until a human fixes the key or URL. 98 workspace tests, clippy clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(changelog): add the watcher and sweeper entry (#640) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop claiming nothing is pulled when a daemon is plainly running `daemonServiceStatus()` asks systemd/launchd and nothing else, so a daemon started by hand — what every developer testing locally has, and what a container has — read as absent. `--connect` and `--status` then told someone whose machine was actively pulling policy and enforcing it that "nothing will be pulled yet", which is simply false and points them at a fix for a problem they do not have. Checks for a live daemon socket before falling back to the service-manager warning. When one is present the advice becomes the true one: install it as a service so it survives reboot and logout. Caught by running the health check in the validation guide against a real demo daemon and reading the output rather than assuming it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [collector] Add the hook-activity source (P1) Ships failproofai's own hook decisions to AgentEye. This is the capability agenteye-collector structurally cannot have: failproofai already sits in the hook path of every supported agent CLI, and nothing else on the machine does. **One source covers every CLI.** The activity store is CLI-agnostic — each row names its own `integration` — so twelve agents are covered by one tailer. Coverage is a function of where hooks are installed, not of anything here. **It maps onto schema AgentEye already has.** `hook_triggered` and `hook_completed` are first-class types with `hook_name`/`hook_id` promoted to columns, a `/hooks` page, and a latency endpoint that pairs the legs on `hook_id`. No server or dashboard work. One activity row carries a duration, so it yields both legs with exact latency rather than an inferred end. **Session ids line up for free**, which is what makes this useful rather than a stream nobody correlates. Verified earlier against the machine's own data: 25 of 43 hook sessions share an exact id with a Claude transcript, and a live Copilot run produced a hook sessionId identical to both its resume id and the id inside its transcript. Agent ids match too — the source derives `<integration>-<project>` from cwd and produced `claude-failproofai`, byte-identical to what the collector's Claude source files the same sessions under. `hook_id` carries the row's byte offset. A per-session id would have collapsed all 8,613 PreToolUse rows of one session into a single row server-side. **Verbosity.** 99.1% of rows are plain `allow`; emitting a pair per row is ~40x the events for no signal. The default keeps every deny and instruct exact and rolls allows up per (session, event, tool, minute), carrying a count so the denominator survives — "we evaluated 19,000 calls and blocked 15" stays answerable. Measured against the real 20,392-row corpus: 7,465 aggregates representing exactly 20,175 allow invocations, plus 168 non-allow completions, matching ground truth to the row. **Cursors are keyed by (dev, inode)**, in a store built to be reused by the tailing engine in the next phase. The activity store rotates by renaming current.jsonl to a page and creating a fresh one; a path-keyed cursor gets both halves wrong at once — re-shipping the rotated page and skipping the new file's first rows. An earlier version of the inode-reuse guard refused to resume whenever the recorded path existed, which is true immediately after every rotation, so it re-shipped every page; it now compares the inode actually at that path. Also fixes a real gap found while verifying: the daemon installed **no tracing subscriber**, so every `tracing::` call in fpai-collect was silently discarded — including the uploader's "the server accepted the request but stored NONE of its events", the single most important signal that a transform is malformed. Known limit, documented in the module: aggregated allow buckets are idempotent only when a re-read covers the same rows. Cursors advance after the spool flush, so a crash reproduces byte-identical buckets that dedup collapses (proven: a second run with cursors intact emits nothing). Losing the cursor file mid-corpus overstates a minute's allow total. Deny and instruct are unaffected — per-row with offset-derived ids. Use verbosity `all` where exact counts matter. 122 workspace tests, clippy clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(changelog): add the hook-activity source entry (#640) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [collector] Add the filetail engine and the Claude source (P2) The generic tailer for append-structured JSONL, plus the first source built on it. Verified against this machine's real transcripts: 5,982 events from 12 sessions across 5 agents, zero warnings. **The engine's invariant** is that every event is a pure function of one line plus its byte offset — nothing folded across a poll window. A live tail that splits a turn across two polls therefore produces byte-identical events to a single full re-read, which is what lets the server's content-hash dedup collapse a re-read instead of storing it twice. Cross-line state lives in the cursor precisely so it holds the same value at the same offset either way. **RereadPolicy exists because two CLIs are not append-only and neither says so.** The probe work found droid rewriting its first line in place when it names a session — shifting every later offset, same inode, mtime restored on a manual rename, so neither inode- nor mtime-watching notices — and cursor rewriting the whole file on the first write of every turn. `ValidatePrefix` re-reads line 1 each poll and rebases the cursor by the difference. Claude declares `ByteCursor`; the switch is one field if that ever changes. **Claude specifics, each verified against real data:** - The agent id comes from the `cwd` field, never from the directory name. Claude encodes cwd by replacing every `/` with `-`, and folder names contain `-` too, so the encoding is not invertible — 3 of 16 project directories on this machine decode wrongly. Live proof: `/home/sidd/Desktop/openclaw-local` produced `claude-openclaw-local`, where splitting the folder name on its last `-` would have given `local` under a parent `openclaw`. - Tool names are carried from the call to its result. A result line names no tool, and the server builds a result row's summary from the name alone, so without this every result is a blank row. Measured: 2,358 of 2,358 result rows carry one. - Token usage is attributed once per message id. One API response spans several lines that each repeat the SAME usage object, so per-line counting multiplies totals several-fold. - Metadata records are skipped by having no timestamp rather than by a type allowlist, so a new record type in a future release costs nothing. Real transcripts carry mode / file-history-snapshot / ai-title / last-prompt. - `/compact` can shrink a transcript; the engine re-reads rather than seeking past EOF, and the server dedups the re-shipped prefix. - Discovery excludes three siblings that each break something different: `.tool-calls.json` is rewritten in place, `journal.jsonl` has a different schema, and `subagents/**` belongs to a format that does not exist yet — claiming it here would ship every subagent line twice under two sessio…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two audit adapters were discarding data their agents do emit. Both were found by installing the CLI, driving a real session against a live provider, and comparing what landed on disk against what the parser produced.
Neither is a regression — both have been wrong since the adapter was written, silently, because nothing asserted on a tool-using pi transcript or a cwd-bearing Hermes session.
pi dropped every tool event
lib/pi-sessions.tshandled onlytextandthinkingcontent blocks.toolCallblocks fell through to the generic"system"branch, and the separaterole: "toolResult"records were never attached to anything — so pi contributed zero tool events to the audit path.The file's own header explained this as "tool-call blocks are not yet observed", and an unused
formatTimestampimport was kept alive with avoidfor "once Pi emits it". The gap was known; the premise behind it was wrong rather than merely stale.Verified against pi 0.73.1 and 0.83.0, driven against a live provider:
@mariozechner/pi-coding-agent(0.73.1) and@earendil-works/pi-coding-agent(0.83.0) packages, so one parser covers both. 0.83.0 emits leading prose alongside the calls where 0.73.1 emitted only calls, so assistant content is no longer assumed homogeneous.hermes contributed nothing to any cwd-scoped audit
listHermesTranscriptMetadataopened with:on the premise that Hermes sessions are gateway sessions and therefore have no working directory. So
failproofai audit --project <repo>reported zero Hermes findings for a repo the user had actually driven Hermes in — no error, no warning, Hermes simply was not there.Verified against hermes-agent 0.19.0: the
sessionstable carries realcwd,git_branchandgit_repo_rootcolumns, and everysource='cli'session populated them (5/5 in the probe).Both shapes are real, so both are now handled:
source='cli'(has a cwd)(profile, source)bucket — unchangedThe data was already present:
HermesSessionRef.cwdwas populated and the SQL already selecteds.cwd. Only the adapter discarded it.Also corrects the goose adapter's docstring, which cited Hermes as the cwd-less counterexample.
Behaviour change worth calling out
A Hermes
source='cli'session'sprojectNamemoves fromhermes:<profile>:clito its encoded working directory, so it groups with the repo it ran in rather than in a Hermes-only bucket. That is the point of the fix, but it will visibly move existing sessions in the dashboard. Gateway sessions are untouched.Tests
__tests__/lib/pi-sessions.test.tsbuilds a real pi transcript from the captured record shapes;__tests__/audit/hermes-adapter-cwd.test.tsbuilds a real SQLite DB (bundled sql.js) holding two cwd-bearing CLI sessions and one cwd-less gateway session.Nine of the new assertions fail against the previous code (5 pi, 4 hermes); the rest are regression guards on behaviour that was already correct — notably that gateway sessions keep their existing bucket and that
hermes://transcript paths are unchanged.Full suite: 2,511 passed, 1 skipped, 146 files.
tsc --noEmitandeslintclean.🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Documentation