Summary
Add AgentCore Runtime as a first-class --sandbox agentcore backend alongside Docker, Daytona, and Modal. AgentCore provides managed, isolated microVMs with shell access via HTTP and WebSocket APIs that map directly to BenchFlow's sandbox contract.
Motivation
- No infrastructure to manage — AgentCore provisions and isolates microVMs on-demand. No servers to run, no SSH keys to rotate.
- Native AWS integration — IAM-based access control, CloudTrail auditing, VPC endpoints, EFS/S3 filesystem mounts.
- Independent scaling — Each runtime is an isolated microVM; parallelism is bounded by service quotas rather than a shared cluster.
- Persistent session storage — Built-in session filesystem that survives stop/resume cycles, enabling warm-start patterns for multi-trial experiments.
Why AgentCore is compatible with BenchFlow
AgentCore exposes three APIs that cover BenchFlow's sandbox needs:
| BenchFlow Requirement |
AgentCore API |
| Run a command, get stdout/stderr/exit_code |
InvokeAgentRuntimeCommand — streams contentStart → contentDelta → contentStop with exitCode over HTTP/2 |
| Hold a long-lived bidirectional stdin/stdout pipe to an agent process |
InvokeAgentRuntimeCommandShell — persistent interactive WebSocket terminal with reconnection support |
| Stop a session and release resources |
StopRuntimeSession — immediately terminates the microVM |
File transfer (upload/download) is handled via base64-over-exec for small files, or S3 Files / EFS mounts for larger payloads.
Proposed Usage
# Install the optional dependency
pip install 'benchflow[sandbox-agentcore]'
# Configure
export BENCHFLOW_AGENTCORE_RUNTIME_ARN="arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/my-agent"
export BENCHFLOW_AGENTCORE_REGION="us-west-2"
# Run
bench eval run \
--sandbox agentcore \
--agent claude-agent-acp \
--model claude-sonnet-4-20250514 \
--tasks-dir my-tasks/Implementation Notes
AgentCoreSandbox(BaseSandbox) — maps exec(), upload_file(), download_file(), start(), stop() to AgentCore APIsAgentCoreProcess(LiveProcess) — maps readline()/writeline() to the WebSocket shell for ACP JSON-RPC transport- Container images must be
linux/arm64 and should run as root (AgentCore's InvokeAgentRuntimeCommand does not invoke a shell by default — commands need bash -c wrapping)
- No snapshot/restore support (
supports_snapshot = False)
off_box_model = True — AgentCore sandboxes are remote microVMs that cannot reach the orchestrator's host network, so the LiteLLM usage-tracking proxy runs inside the sandbox (same as Daytona and Modal)
Known Limitation
The bedrock-agentcore Python SDK's open_shell() currently rejects harness ARNs (only accepts runtime/ ARNs). A standalone runtime (not harness-managed) is required. See related SDK bug: open_shell() rejects harness ARNs that InvokeAgentRuntimeCommand accepts.
References
Summary
Add AgentCore Runtime as a first-class
--sandbox agentcorebackend alongside Docker, Daytona, and Modal. AgentCore provides managed, isolated microVMs with shell access via HTTP and WebSocket APIs that map directly to BenchFlow's sandbox contract.Motivation
Why AgentCore is compatible with BenchFlow
AgentCore exposes three APIs that cover BenchFlow's sandbox needs:
InvokeAgentRuntimeCommand— streamscontentStart→contentDelta→contentStopwithexitCodeover HTTP/2InvokeAgentRuntimeCommandShell— persistent interactive WebSocket terminal with reconnection supportStopRuntimeSession— immediately terminates the microVMFile transfer (upload/download) is handled via base64-over-exec for small files, or S3 Files / EFS mounts for larger payloads.
Proposed Usage
Implementation Notes
AgentCoreSandbox(BaseSandbox)— mapsexec(),upload_file(),download_file(),start(),stop()to AgentCore APIsAgentCoreProcess(LiveProcess)— mapsreadline()/writeline()to the WebSocket shell for ACP JSON-RPC transportlinux/arm64and should run as root (AgentCore'sInvokeAgentRuntimeCommanddoes not invoke a shell by default — commands needbash -cwrapping)supports_snapshot = False)off_box_model = True— AgentCore sandboxes are remote microVMs that cannot reach the orchestrator's host network, so the LiteLLM usage-tracking proxy runs inside the sandbox (same as Daytona and Modal)Known Limitation
The
bedrock-agentcorePython SDK'sopen_shell()currently rejects harness ARNs (only acceptsruntime/ARNs). A standalone runtime (not harness-managed) is required. See related SDK bug: open_shell() rejects harness ARNs that InvokeAgentRuntimeCommand accepts.References