Problem
The Bash script generated by WrapShellScript in cli/internal/raylogoffload/offload.go can discard a termination signal received after its signal trap is installed but before it records the background workload PID.
The signal handler only forwards TERM when tau_driver_child is nonempty. The workload can run and publish readiness before the parent executes tau_driver_child="$!". A TERM delivered in that interval is handled without forwarding or recording it; the parent subsequently waits for a workload that never received the termination request.
Observed CI failure
PR #297 validation failed in TestWrapShellScriptForwardsTermToWorkload:
--- FAIL: TestWrapShellScriptForwardsTermToWorkload (2.00s)
offload_test.go:480: wrapped workload did not receive TERM within 2s
Failing job: https://github.com/Azure/taugrid/actions/runs/35613761558/job/106379133213?pr=297#step:15:35
The downstream validation summary also failed because GO_RESULT=failure; that is a consequence of the Go test failure, not a separate shell configuration error.
Evidence boundary: the startup race was reproduced locally with a controlled scheduling barrier and produces the same timeout. The CI log does not record the actual process scheduling, so it does not conclusively establish that this race caused that particular CI failure. The original focused test also passed 50 consecutive local runs before the controlled reproduction.
Controlled reproduction
- Generate a wrapper around a workload that installs a TERM trap exiting with code 42, publishes readiness, and keeps running.
- Insert a test-only barrier immediately before the parent captures the child PID.
- Wait until both the workload and parent barrier are ready, then send SIGTERM to the wrapper.
- Release the barrier from the signal handler after calling the production forwarding function.
- Assert that the wrapper exits with code 42 and writes the matching completion marker.
Without preserving the early signal, the startup case times out. The barrier makes the signal arrive while the PID variable is empty instead of depending on scheduler timing.
Expected behavior and impact
Once the wrapper has installed its termination handler, a signal arriving before PID registration should be retained and forwarded when the PID becomes available. The wrapper should terminate the workload process group, preserve the workload exit status, and write its completion marker.
Losing the signal can leave the workload running and the wrapper waiting until an external timeout or forced termination. Increasing the test timeout does not address the lost signal.
Locally validated fix approach
- Record a pending signal before checking whether the child PID is available.
- After capturing the child PID, forward a pending signal if no forwarded-child exit status has been recorded.
- Preserve existing process-group forwarding, exit-code handling, and completion-marker behavior.
- Extend the existing regression test with the controlled startup case and clean up workload processes on failure.
Local validation of the patch:
- Controlled startup regression failed before the production fix and passed afterward.
- Shell-wrapper tests passed 50 repetitions with the Go race detector (the deterministic barrier, not the Go race detector, checks the Bash scheduling race).
- Full CLI tests, go vet, formatting checks, Staticcheck v0.7.0, and license-header checks passed.
- The two-second test timeout was not increased.
A new GitHub CI result for this fix has not been verified. This issue records the defect and local evidence; it does not claim that the fix has merged.
Problem
The Bash script generated by
WrapShellScriptincli/internal/raylogoffload/offload.gocan discard a termination signal received after its signal trap is installed but before it records the background workload PID.The signal handler only forwards TERM when
tau_driver_childis nonempty. The workload can run and publish readiness before the parent executestau_driver_child="$!". A TERM delivered in that interval is handled without forwarding or recording it; the parent subsequently waits for a workload that never received the termination request.Observed CI failure
PR #297 validation failed in
TestWrapShellScriptForwardsTermToWorkload:Failing job: https://github.com/Azure/taugrid/actions/runs/35613761558/job/106379133213?pr=297#step:15:35
The downstream validation summary also failed because
GO_RESULT=failure; that is a consequence of the Go test failure, not a separate shell configuration error.Evidence boundary: the startup race was reproduced locally with a controlled scheduling barrier and produces the same timeout. The CI log does not record the actual process scheduling, so it does not conclusively establish that this race caused that particular CI failure. The original focused test also passed 50 consecutive local runs before the controlled reproduction.
Controlled reproduction
Without preserving the early signal, the startup case times out. The barrier makes the signal arrive while the PID variable is empty instead of depending on scheduler timing.
Expected behavior and impact
Once the wrapper has installed its termination handler, a signal arriving before PID registration should be retained and forwarded when the PID becomes available. The wrapper should terminate the workload process group, preserve the workload exit status, and write its completion marker.
Losing the signal can leave the workload running and the wrapper waiting until an external timeout or forced termination. Increasing the test timeout does not address the lost signal.
Locally validated fix approach
Local validation of the patch:
A new GitHub CI result for this fix has not been verified. This issue records the defect and local evidence; it does not claim that the fix has merged.