Add harness for Hermes Agent - #178
Closed
eugeneng04 wants to merge 6 commits into
Closed
Conversation
This introduces the agent harness for Hermes, which executes local agents and parses trajectory states via the local SQLite database. Also includes tests for sqlite parsing and configuration overlaying. TAG=agy CONV=80659b55-e197-42ab-a6a0-e89f6b649020
Refactored the Hermes CLI agent harness: - Replaced standard subprocess with devops_run core wrapper. - Routed provider mappings through centralized resolve_provider. - Removed redundant custom provider alias functions and api_key spraying logic. - Standardized file paths and constants (e.g. _SOUL_FILE). - Improved trajectory parsing by removing database session limit/sort constraints since sessions are run-scoped in temp directory. - Handled edge cases like missing tool names gracefully. Fixed unit test environment isolation: - Fixed test_models_claude.py by clearing os.environ inside the mocked async tests to prevent local environment variables (like AGENT_API_KEY) from forcing first-party client initialization. - Fixed test_agents_cli_openclaw.py by using monkeypatch to delete API keys before asserting that they are absent from the execution environment. Updated agents component documentation to include the hermes agent in the supported harnesses registry. TAG=agy CONV=0607f41a-7efc-43f2-81a8-8219534f7b03
pradeepvrd
reviewed
Jul 13, 2026
The hermes harness reported no token usage at all (AgentResult.tokens defaulted to {}). Hermes persists per-session counts in state.db's sessions table (input_tokens, output_tokens, reasoning_tokens, cache_read_tokens, cache_write_tokens), populated by its per-turn cost calculation. Read the latest session's counts into the canonical six-bucket dict (input, cached, cache_write, reasoning, output, total) with None for unreported buckets — never a fabricated 0. Columns are probed via PRAGMA table_info so older Hermes schemas degrade to all-None, as do missing/corrupt DBs; the timeout path reports whatever was flushed before the kill.
Collaborator
Author
|
Added token usage reporting ( Notes for review:
|
- Sum token counts across all sessions instead of 'latest by id' — the DB is per-run, session ids are TEXT (lexicographic order is not chronological), and multi-session runs were silently undercounted. Also replaces the PRAGMA column probe + interpolated column list with a static SUM select (old schemas fall through the existing sqlite3.Error handler).
- Timeout path: token extraction moved out of the trajectory try/except (it never raises), so a trajectory failure no longer skips tokens and token errors can't be mislabeled as trajectory errors.
- Binary-unavailable path now returns the same canonical all-None token shape as every other path (was {}).
- Coerce REAL-affinity counts instead of dropping them to None; shared read-only connect for both DB readers (the trajectory reader previously opened read-write, which can lock a live DB).
- Add agent-level tests for the tokens= wiring (success + binary-unavailable), sum/REAL/NULL parsing tests, dedup the test schema fixture.
- Docs: agents.md hermes row mentions token accounting; known_issues.md documents the unverified assumption that Hermes normalizes usage to Anthropic-style buckets (if provider-native, total double-counts cached/reasoning).
Upstream main's AgentHarness.run now passes workspace_path positionally to _execute (and wraps it in tracing that binds the signature), so the old _execute(self, prompt) signature broke the PR's merge CI with 'too many positional arguments' — surfaced by the new agent-level token tests, which are the first tests on this branch to call run(). Accept the parameter like every other harness and use it as the run dir when supplied; the shared agent_workdir helper does not exist on this branch's base, so the tempdir fallback stays until a rebase.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for the Hermes CLI agent as a first-class agent harness in
devops-bench.Hermes is an autonomous DevOps agent CLI built by Nous Research. This integration enables running
the benchmark against Hermes using standard models and providers (including GCP Vertex AI and Google
GenAI), extracting detailed execution trajectories, and running end-to-end evaluations.
Key Additions & Refactorings
Hermes Agent Harness (
devops_bench/agents/cli/hermes/)HermesAgentclass, which stages agent configuration,.envfile context, andSOUL.mdin a per-run isolated temp directory ($HERMES_HOME) to preventstate leakage.
devops_runcore process wrapperwith custom environment overlays and timeout configuration.
resolve_providerto map thegoogle-vertexprovider onto the--provider vertexbackend.SQLite
state.dbdatabase. Trajectory retrieval is optimized since sessions are run-scoped inside atemporary directory.
Integration & Docs
_BUILTIN_AGENT_MODULESindevops_bench/evalharness/default.pytosupport lazy module loading.
docs/components/agents.mdto addhermesin the supported harnesses table, detailingits execution model, SQLite database storage pattern, and structural capabilities (MCP, skills, rules).
Test Coverage
(
test_hermes_agent.py).test_hermes_parsing.py) evaluating SQLiteschema queries, tool arguments parsing, missing attributes, and session queries.
Environment Isolation Fixes
os.environintest_models_claude.pytests toprevent parent environment credentials (e.g.
AGENT_API_KEY) from bleeding into mock tests andforcing real Anthropic client initialization.
test_agents_cli_openclaw.pyto prune API keysfrom the subprocess test env to verify keyless/ADC authentication paths correctly.
Testing Done
uv run pytest(all 895 tests passed).hermes-test-clusterin GCP projectngeugene-gke-dev.