Skip to content

Add setup --cli claude --verify static hook check - #283

Merged
Siddhant-K-code merged 2 commits into
mainfrom
siddhant-k-code-claude-code-setup-verify
Oct 22, 2026
Merged

Siddhant-K-code merged 2 commits into
mainfrom
siddhant-k-code-claude-code-setup-verify

Conversation

@Siddhant-K-code

@Siddhant-K-code Siddhant-K-code commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Refs #239

What changed

agent-strace setup --cli claude --verify now runs a read-only, static check of the Claude Code hooks. It matches setup --cli codex --verify (#264, #275). Before this change, --verify with --cli claude exited 2.

  • setup_verify.py: the hook checks (event, command, type, matcher, duplicates, stale hooks, redaction mode) and the executable check are now shared helpers. The Codex verifier uses them, and its messages are unchanged. The new verify_claude_setup adds the Claude Code checks.
  • cli.py: cmd_setup sends --cli claude --verify to the new check. New helpers _claude_config_dir, _claude_settings_target and _claude_managed_settings_dir. setup --cli claude and --cli all name $CLAUDE_CONFIG_DIR/settings.json when that variable is set, so setup and --verify point at the same file. New --verify and --global help text. The error for other providers now names both supported providers.
  • Docs: a new "Claude Code hook verification" section in docs/setup.md. Updated the setup paragraph and the --verify row in docs/commands.md, the verify step in docs/v1-workflow.md, and one README quick start line. The capture matrix does not describe setup verification, so it is unchanged.
  • 27 new tests in tests/test_setup_verify.py. TestCodexSetup in tests/test_codex_hooks.py now clears CLAUDE_CONFIG_DIR, so its setup output check does not depend on the shell.

The check exits 1 with a message that names the file and the fix when:

  • the user settings file is missing, is not valid JSON, or has no AgentTrace hooks;
  • a hook is missing, stale, under the wrong event, duplicated, not a command handler, or narrowed by its matcher;
  • an AgentTrace hook has an if or args field, which stops it from running as generated;
  • the hooks use mixed redaction modes;
  • a settings layer turns hooks off;
  • agent-strace on PATH is missing or reports another version.

On success it prints the user settings path, the event count, the redaction mode, each layer it read, the executable, a "Not checked:" line, and a note that a pass does not prove capture. This keeps "config written" apart from "events observed", as #239 asks. Telemetry already tags these runs subcommand=verify with integration=claude.

Decisions

  1. Settings layers.
    • User settings are required, because setup targets them. That is $CLAUDE_CONFIG_DIR/settings.json when the variable is set, otherwise ~/.claude/settings.json.
    • Claude Code reads path.resolve(CLAUDE_CONFIG_DIR ?? join(homedir(), ".claude"))/settings.json. I checked this in the Claude Code 2.1.222 bundle. So a set value is used as is: ~ is not expanded, a relative path starts from the current directory, and an empty value means the current directory. The check resolves it the same way, and setup names the same file.
    • Project .claude/settings.json and local .claude/settings.local.json in the current directory are read when present. Claude Code reads the shared project file from the session's working directory.
    • Managed managed-settings.json and managed-settings.d/*.json are read when present, in the documented order. Hidden and non .json files are ignored. A managed file that the OS will not let us read is skipped, as Claude Code skips it.
    • Not checked, and listed in the output: other settings values (run claude doctor), the --settings, --setting-sources, --bare and --safe-mode flags, MDM and server-managed policy, plugin and skill hooks, and settings.local.json at the repository root when Claude Code starts in a subdirectory or worktree. Only CLAUDE_CONFIG_DIR from the shell is used. A value set in a settings env block is not followed.
  2. Settings that stop the hooks fail, like Codex [features].hooks = false in Fail setup --cli codex --verify when config.toml disables hooks #275.
    • disableAllHooks: true fails when it is in effect. Managed wins, then local, then project, then user.
    • disableAllHooks: true in user settings fails even when a project or local file sets it back to false, because other projects then run no hooks.
    • A disableAllHooks value that is not true or false fails in user, project or local settings. In managed settings it is ignored, because Claude Code drops it.
    • Managed allowManagedHooksOnly fails unless it is false or "false". Claude Code reads an invalid value as true. It also fails when managed settings hold AgentTrace hooks, because setup targets user settings and this repo documents no managed install.
    • Across managed files, a later value replaces an earlier one. A null leaves the earlier value in place. The docs say a null removes the key and do not count it as setting the key. This choice can only fail more, never pass more.
  3. AgentTrace hooks in project, local or managed settings.
    • Exact copies of the user settings handlers pass, because Claude Code runs the same handler once when more than one file defines it. Matchers are not compared.
    • Any other AgentTrace hook there fails, because it can record events twice or with another redaction mode. The message names the file and the events.
    • Hooks only in project settings fail on the missing user hooks. A second message says where the hooks are.
  4. Invalid JSON fails in every file the check reads, as Codex does. Claude Code reports a settings error and does not apply a broken user, project or local file. It refuses to start with a broken managed file. An empty file reads as {}.
  5. --cli all, gemini, cursor and copilot still exit 2 and write nothing. A run that checked only some providers could look like a full pass. The message is now "Setup verification supports only --cli codex or --cli claude, one provider per run."
  6. Matchers. PreToolUse, PostToolUse, PostToolUseFailure, SessionStart and SessionEnd need a match-all matcher ("", "*", ".*" or omitted). Narrower matchers fail. Claude Code ignores matchers on UserPromptSubmit and Stop, so those are not checked. Unrelated user hooks and extra handler fields such as timeout are allowed.

Claude Code docs used

Testing

  • Full suite on Python 3.13 with TMPDIR=/private/tmp DO_NOT_TRACK=1 AGENT_STRACE_TELEMETRY=0 PYTHONPATH=src python3 -m unittest discover -s tests: 2290 tests, OK (skipped=1). The baseline on 5d5004b is 2263, so this adds 27 tests.
  • tests.test_setup_verify and tests.test_codex_hooks on Python 3.10.20: 79 tests, OK (skipped=1).
  • Manual CLI probes with a temp HOME and project directory. Missing file, invalid JSON, missing hook, stale hook, disableAllHooks in user, project and local settings all exited 1 with a fix. Unrelated user hooks next to ours and a clean install exited 0. --cli all exited 2. No probe wrote a file.
  • CLAUDE_CONFIG_DIR probes. Set to ~/ccx or empty, hooks under $HOME exited 1 and named the file Claude Code reads, and hooks at the path that setup printed exited 0. Setup named $CLAUDE_CONFIG_DIR/settings.json for --cli claude and --cli all, and ~/.claude/settings.json when the variable is unset.
  • Mutation check of the new resolution and setup output lines: 7 of 7 mutants fail the tests.
  • Not run against a live, authenticated Claude Code session. Add verified Copilot CLI capture-matrix entry and verify Codex #280 deferred that too.

Rebased on 5d5004b (#281). docs/commands.md keeps the #281 Codex wording and adds the Claude parts. No version bump.

Siddhant-K-code and others added 2 commits October 22, 2026 21:54
Read-only check of the AgentTrace hooks in Claude Code user settings,
plus the project, local and managed settings that can turn them off.
The hook and executable checks are shared with the Codex verifier.

Refs #239

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Claude Code reads user settings from
path.resolve(CLAUDE_CONFIG_DIR ?? join(homedir(), ".claude"))/settings.json.
A set value is used as is: "~" is not expanded, and an empty value resolves
to the current directory. --verify now resolves the directory the same way,
so it no longer passes on a file Claude Code never reads.

setup --cli claude (and --cli all) now names $CLAUDE_CONFIG_DIR/settings.json
when that variable is set, so setup and --verify point at the same file.

Refs #239

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@Siddhant-K-code
Siddhant-K-code merged commit e5bce81 into main Oct 22, 2026
5 checks passed
@Siddhant-K-code
Siddhant-K-code deleted the siddhant-k-code-claude-code-setup-verify branch October 22, 2026 16:52
Siddhant-K-code added a commit that referenced this pull request Oct 22, 2026
Bump 0.99.0 to 0.100.0. Includes #280 (verified Copilot CLI capture-matrix entry, Codex verified), #281 (setup: resolve the Codex directory the way Codex does), #282 (health: report event types and redaction state) and #283 (setup --cli claude --verify).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant