Skip to content

fix(compute): allow stopping errored sandboxes for recovery - #4231

Open
shiju-nv wants to merge 2 commits into
mainfrom
fix/sandbox-start-failure-recovery-3308
Open

shiju-nv wants to merge 2 commits into
mainfrom
fix/sandbox-start-failure-recovery-3308

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

An infrastructure startup failure can leave a retained sandbox in Error, with both stop and start refusing recovery even after the underlying problem is fixed. Allow explicit stop from that state so operators can use the existing start path without deleting the sandbox or its persistent workspace. Provisioning timeouts keep their existing cleanup-then-start recovery path.

Related Issue

Closes #3308

Changes

  • Accept ordinary Error states in stop, while preserving pending-operation and timeout cleanup checks.
  • Tell direct-start callers to stop the sandbox before retrying.
  • Cover startup-failure recovery, renewed authentication and configuration admission, driver error reasons, pending operations, timeout cleanup, and failed-stop retries.
  • Document the recovery commands and the provisioning-timeout exception.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated
  • E2E tests added/updated (local Docker reproducer executed; no new repository E2E test)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (sandbox lifecycle documentation)

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Signed-off-by: Shiju <shiju@nvidia.com>
Remove duplicate cancellation and idempotency checks covered by existing lifecycle tests. Keep recovery guidance focused on the stop/start commands and timeout exception.

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv
shiju-nv force-pushed the fix/sandbox-start-failure-recovery-3308 branch from e1b2d1a to 4c7bde5 Compare October 7, 2026 15:18

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sandbox stuck in unrecoverable Error phase after supervisor session drops (SSH disconnect); stop/start refuse to act

1 participant