Kill orphaned prover processes on Linux when tlapm dies - #284
Open
tangruize wants to merge 1 commit into
Open
Conversation
When tlapm is SIGKILLed or OOM-killed it cannot run its usual cleanup, so the prover it spawned (e.g. z3) is reparented to init and keeps consuming CPU and memory, unmanaged by tlapm, until it happens to terminate on its own. Run each prover via `exec setpriv --pdeathsig KILL` on Linux so the kernel SIGKILLs it when tlapm dies. Injected at the single choke point get_exec, so it covers all single-command backends. Timeout semantics are unchanged; on non-Linux, or when setpriv is missing or too old to support --pdeathsig, the prefix is empty and behaviour is identical. Signed-off-by: Ruize Tang <1466040111@qq.com>
Member
|
This PR is Linux-specific, which is completely reasonable. That said, I’m seeing a similar issue on macOS (when using AI to drive TLAPS). It would be great to address the same problem on macOS and Windows in a future follow-up. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
tlapm launches every backend prover (z3, cvc4, zenon, …) as a child process and terminates it as part of its own cleanup. If tlapm dies without getting to run that cleanup — an uncatchable
SIGKILL, or an OOM-kill under memory pressure — the prover it spawned is reparented toinitand keeps running. Most solvers impose no limit on themselves, so the orphan goes on consuming CPU and memory until it happens to finish on its own.Fix
On Linux, run each prover through util-linux
setpriv --pdeathsig KILL, which arms the kernel parent-death signal (PR_SET_PDEATHSIG): the kernel sendsSIGKILLto the prover the moment its parent (tlapm) dies, including theSIGKILLand OOM cases where tlapm can no longer act itself.The leading
execmakes the prover replace the shell that launches it, so the prover becomes tlapm's direct child — the process the parent-death signal actually watches — instead of a grandchild under an intermediate/bin/sh. Without it that shell would stay alive and keep the prover running.The prefix is added at the single point where every backend command string is assembled, so all provers are covered at once. On non-Linux, or on Linux where
setprivis missing or too old to support--pdeathsig, the prefix is empty and behaviour is unchanged.