Skip to content

fix(runtime): cancel WASM watchdog on early exit - #1278

Open
andyst-dev wants to merge 1 commit into
RightNow-AI:mainfrom
andyst-dev:fix/1241-wasm-watchdog-cancel
Open

fix(runtime): cancel WASM watchdog on early exit#1278
andyst-dev wants to merge 1 commit into
RightNow-AI:mainfrom
andyst-dev:fix/1241-wasm-watchdog-cancel

Conversation

@andyst-dev

Copy link
Copy Markdown

Fixes #1241

Summary

sandbox.rs spawns a detached OS thread per WASM execution as an epoch watchdog: it sleeps the full timeout_secs and then bumps the shared engine epoch. When execution finishes early (success, fuel exhaustion, or error), the thread keeps sleeping and eventually calls increment_epoch() on the shared engine — which can interrupt a different execution still running on the same engine.

Beyond the reported thread accumulation under load, this is a cross-execution correctness bug: a stale watchdog can cut a concurrent execution short with a false "timed out" error.

Changes

  • The watchdog thread now polls a cancellation flag while sleeping (50ms steps) and exits immediately when execution has finished.
  • A small RAII guard (WatchdogCancel) sets the flag on drop, so cancellation is guaranteed on every exit path of execute_sync — success, fuel exhaustion, epoch interrupt, or any error.
  • Added a regression test that runs a long loop and a fast execution concurrently on the same engine: the fast execution's stale watchdog would previously bump the epoch at t=1s and interrupt the long execution early. With the fix, the long execution is only interrupted by its own watchdog at its full timeout.

The stale-watchdog behavior is directly observable: before the fix the test fails with long execution was interrupted after 1.05s; after the fix it completes at its own 4s timeout.

Testing

  • cargo test -p openfang-runtime — all tests pass (sandbox suite: 89 passed)
  • cargo fmt -p openfang-runtime -- --check — clean
  • cargo clippy -p openfang-runtime --lib -- -D warnings — no warnings on the changed crate

Security

  • No new unsafe code
  • No secrets or API keys in diff
  • User input validated at boundaries

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: WASM watchdog threads accumulate under load — not cancelled on early exit

1 participant