Skip to content

Add harness for Hermes Agent - #178

Closed
eugeneng04 wants to merge 6 commits into
gke-labs:mainfrom
eugeneng04:hermes-harness-refactor
Closed

Add harness for Hermes Agent#178
eugeneng04 wants to merge 6 commits into
gke-labs:mainfrom
eugeneng04:hermes-harness-refactor

Conversation

@eugeneng04

@eugeneng04 eugeneng04 commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR adds support for the Hermes CLI agent as a first-class agent harness in devops-bench.

Hermes is an autonomous DevOps agent CLI built by Nous Research. This integration enables running
the benchmark against Hermes using standard models and providers (including GCP Vertex AI and Google
GenAI), extracting detailed execution trajectories, and running end-to-end evaluations.

Key Additions & Refactorings

  1. Hermes Agent Harness (devops_bench/agents/cli/hermes/)

    • Harness Execution: Implemented the HermesAgent class, which stages agent configuration,
      .env file context, and SOUL.md in a per-run isolated temp directory ($HERMES_HOME) to prevent
      state leakage.
    • Process Wrapping: Drives the agent using the centralized devops_run core process wrapper
      with custom environment overlays and timeout configuration.
    • Provider Resolution: Integrated the harness with the centralized model contract
      resolve_provider to map the google-vertex provider onto the --provider vertex backend.
    • Trajectory Parser: Extracts the execution trajectory and tool details from the agent's
      SQLite state.db database. Trajectory retrieval is optimized since sessions are run-scoped inside a
      temporary directory.
  2. Integration & Docs

    • Added registration to _BUILTIN_AGENT_MODULES in devops_bench/evalharness/default.py to
      support lazy module loading.
    • Updated docs/components/agents.md to add hermes in the supported harnesses table, detailing
      its execution model, SQLite database storage pattern, and structural capabilities (MCP, skills, rules).
  3. Test Coverage

    • Added unit tests for agent configuration staging, binary resolution, and command-line execution
      (test_hermes_agent.py).
    • Added comprehensive trajectory extraction tests (test_hermes_parsing.py) evaluating SQLite
      schema queries, tool arguments parsing, missing attributes, and session queries.
  4. Environment Isolation Fixes

    • Claude Client Test Protection: Cleared os.environ in test_models_claude.py tests to
      prevent parent environment credentials (e.g. AGENT_API_KEY) from bleeding into mock tests and
      forcing real Anthropic client initialization.
    • OpenClaw Harness Test Protection: Updated test_agents_cli_openclaw.py to prune API keys
      from the subprocess test env to verify keyless/ADC authentication paths correctly.

Testing Done

  • Ran the full test suite locally with uv run pytest (all 895 tests passed).
  • Verified GKE cluster connectivity to hermes-test-cluster in GCP project ngeugene-gke-dev.

This introduces the agent harness for Hermes, which executes local agents and parses trajectory states via the local SQLite database. Also includes tests for sqlite parsing and configuration overlaying.

TAG=agy
CONV=80659b55-e197-42ab-a6a0-e89f6b649020
Refactored the Hermes CLI agent harness:
- Replaced standard subprocess with devops_run core wrapper.
- Routed provider mappings through centralized resolve_provider.
- Removed redundant custom provider alias functions and api_key spraying logic.
- Standardized file paths and constants (e.g. _SOUL_FILE).
- Improved trajectory parsing by removing database session limit/sort constraints since sessions are run-scoped in temp directory.
- Handled edge cases like missing tool names gracefully.

Fixed unit test environment isolation:
- Fixed test_models_claude.py by clearing os.environ inside the mocked async tests to prevent local environment variables (like AGENT_API_KEY) from forcing first-party client initialization.
- Fixed test_agents_cli_openclaw.py by using monkeypatch to delete API keys before asserting that they are absent from the execution environment.

Updated agents component documentation to include the hermes agent in the supported harnesses registry.

TAG=agy
CONV=0607f41a-7efc-43f2-81a8-8219534f7b03
Comment thread devops_bench/agents/cli/hermes/agent.py
Comment thread devops_bench/agents/cli/hermes/agent.py Outdated
Comment thread devops_bench/agents/cli/hermes/parsing.py Outdated
Comment thread devops_bench/agents/cli/hermes/parsing.py Outdated
Comment thread docs/components/agents.md Outdated
Comment thread tests/unit/models/test_models_claude.py Outdated
Comment thread devops_bench/agents/cli/hermes/agent.py
The hermes harness reported no token usage at all (AgentResult.tokens defaulted to {}). Hermes persists per-session counts in state.db's sessions table (input_tokens, output_tokens, reasoning_tokens, cache_read_tokens, cache_write_tokens), populated by its per-turn cost calculation. Read the latest session's counts into the canonical six-bucket dict (input, cached, cache_write, reasoning, output, total) with None for unreported buckets — never a fabricated 0. Columns are probed via PRAGMA table_info so older Hermes schemas degrade to all-None, as do missing/corrupt DBs; the timeout path reports whatever was flushed before the kill.
@eugeneng04

Copy link
Copy Markdown
Collaborator Author

Added token usage reporting (eccfbd2): the harness previously set no tokens at all. It now reads the latest session's counts from state.db's sessions table (input_tokens / output_tokens / reasoning_tokens / cache_read_tokens / cache_write_tokens) into the canonical six-bucket shape — {input, cached, cache_write, reasoning, output, total}, None for unreported buckets.

Notes for review:

  • Hermes runs Anthropic prompt caching with explicit cache_control breakpoints (always-on), so cached — and cache_write, which few harnesses can report — should carry real volume.
  • Columns are probed via PRAGMA table_info, so older Hermes schemas (and missing/corrupt DBs) degrade to all-None, never a fabricated 0. The timeout path reports whatever was flushed before the kill.
  • One assumption to verify against a real state.db: that Hermes's input_tokens excludes cache reads (the Anthropic convention, since the columns mirror Anthropic usage fields). If it turns out to be the full prompt, the mapping needs an input − cache_read adjustment.
  • Aligns with the canonical token schema in feat: canonical token buckets across harnesses and the result row #212 (this branch defines the bucket tuple locally; it can import the shared helper once feat: canonical token buckets across harnesses and the result row #212 lands).

- Sum token counts across all sessions instead of 'latest by id' — the DB is per-run, session ids are TEXT (lexicographic order is not chronological), and multi-session runs were silently undercounted. Also replaces the PRAGMA column probe + interpolated column list with a static SUM select (old schemas fall through the existing sqlite3.Error handler).
- Timeout path: token extraction moved out of the trajectory try/except (it never raises), so a trajectory failure no longer skips tokens and token errors can't be mislabeled as trajectory errors.
- Binary-unavailable path now returns the same canonical all-None token shape as every other path (was {}).
- Coerce REAL-affinity counts instead of dropping them to None; shared read-only connect for both DB readers (the trajectory reader previously opened read-write, which can lock a live DB).
- Add agent-level tests for the tokens= wiring (success + binary-unavailable), sum/REAL/NULL parsing tests, dedup the test schema fixture.
- Docs: agents.md hermes row mentions token accounting; known_issues.md documents the unverified assumption that Hermes normalizes usage to Anthropic-style buckets (if provider-native, total double-counts cached/reasoning).
Upstream main's AgentHarness.run now passes workspace_path positionally to _execute (and wraps it in tracing that binds the signature), so the old _execute(self, prompt) signature broke the PR's merge CI with 'too many positional arguments' — surfaced by the new agent-level token tests, which are the first tests on this branch to call run(). Accept the parameter like every other harness and use it as the run dir when supplied; the shared agent_workdir helper does not exist on this branch's base, so the tempdir fallback stays until a rebase.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants