In aspect_rules_js we have a js_run_binary macro that defaults to silent_on_success = True, meaning that we capture the stdout and stderr of the actual executable to temporary files and only print them out upon failure. As a result, we usually have two processes: a parent bash process waiting for its child process (node) to complete.
We heard a report of this causing some non-deterministic build behavior, and it appears that this is what happened:
- The build action exceeded its timeout and so Buildbarn sent the parent process a
SIGKILL.
- This killed the parent process immediately but left the orphan
node process running.
- The same build action was scheduled again and landed on the same machine using the same path.
- The orphan process and the new build action ended up writing concurrently to the same output files.
I got an AI bot to write some unit tests and a possible fix on the branch here. The solution it came up with was to put each build action in its own process group so that we can cancel the whole group at once.
In aspect_rules_js we have a
js_run_binarymacro that defaults to silent_on_success = True, meaning that we capture the stdout and stderr of the actual executable to temporary files and only print them out upon failure. As a result, we usually have two processes: a parentbashprocess waiting for its child process (node) to complete.We heard a report of this causing some non-deterministic build behavior, and it appears that this is what happened:
SIGKILL.nodeprocess running.I got an AI bot to write some unit tests and a possible fix on the branch here. The solution it came up with was to put each build action in its own process group so that we can cancel the whole group at once.