Skip to content

Add Claude Code CLI as a ringer engine - #135

Open
hcoronel1 wants to merge 1 commit into
NateBJones-Projects:mainfrom
hcoronel1:pr-claude-engine
Open

hcoronel1 wants to merge 1 commit into
NateBJones-Projects:mainfrom
hcoronel1:pr-claude-engine

Conversation

@hcoronel1

Copy link
Copy Markdown

Summary

  • Adds the claude/claude-sonnet engine identity and model registry entries (claude-sonnet-5, claude-opus-5, claude-fable-5-1, claude-haiku-4-5, alias sonnet) so Claude Code CLI can run as a worker engine alongside Codex, Grok Build, and OpenCode/OpenRouter.
  • Adds result_parser = "claude-code-json" and parse_claude_code_json_result() to parse the claude -p --output-format json result envelope: real token usage from usage, and the effective model from modelUsage (Claude Code can resolve an alias like sonnet to a dated model id).
  • Distinguishes transport/API failures (rate limits, timeouts — marked scoreable=False, failure_kind="infrastructure", excluded from the scoreboard, no verify/retry) from agent-level failures (max turns, tool/permission rejection — still scoreable=True, so a genuine quality loss isn't hidden from the scoreboard). An earlier version of this PR classified every is_error as infrastructure; fixed after a Codex read-only audit caught it.
  • Reads the JSON result structurally — one JSON object per line, or the whole trimmed payload as a single object — instead of scanning the text for a bare {. The earlier bracket-scan could match a { an agent happened to quote inside its own result text and misattribute tokens/model/error to that embedded fragment. Also fixed after the same audit.
  • _claude_reported_model only returns a family match (sonnet/opus/fable/haiku) when every same-family candidate in modelUsage agrees on the same canonical model; otherwise it returns None instead of guessing the first one. Guards against misattributing identity when a director and a same-family subagent both appear (e.g. two different Sonnet versions).
  • Schema v3→v4 migration (scoreable, failure_kind columns) with a backward-compatible ALTER TABLE.

Motivation

I wanted to run real production tasks through Ringer using Claude Code CLI as the worker, and needed the scoreboard to stay honest about why a run failed — an API rate limit isn't the model's fault, but Ringer's existing token/model regex parsers had no way to tell an infra failure apart from a real one for a JSON-emitting engine.

Test plan

  • python3 -m pytest -q — 278/278 passed (was 275 before this branch's own fixes)
  • New coverage in tests/test_claude_engine.py: JSON result selection (including the embedded-JSON-in-string case), infra vs. agent error classification, ambiguous same-family model attribution, schema migration
  • Self-reviewed with a Codex read-only audit before opening this PR; findings incorporated (see bullets above)

🤖 Generated with Claude Code

Adds the claude/claude-sonnet engine identity, model registry entries,
and a claude-code-json result parser for the `claude -p --output-format
json` envelope.

The parser distinguishes transport/API failures (rate limits, timeouts —
excluded from the scoreboard as non-scoreable "infrastructure", no
verify/retry) from agent-level failures (max turns, tool/permission
rejection — still scoreable, so a real quality loss isn't hidden from
the scoreboard). It reads the JSON result structurally (whole-payload
decode, one JSON object per line) instead of scanning for a bare `{`,
so text an agent quotes inside its own result can't be mistaken for the
envelope. Model attribution from `modelUsage` only returns a family
match when every candidate of that family agrees on the same canonical
model, rather than guessing the first one when a director and a
same-family subagent both appear.

Adds a schema v3->v4 migration (scoreable, failure_kind columns) with a
backward-compatible ALTER TABLE.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants