Repository navigation
Stop mp_prefetch workers when the parent process dies - #1475
Open
princemittalr wants to merge 1 commit into
Open
princemittalr wants to merge 1 commit into
princemittalr wants to merge 1 commit into
Conversation
Prefetch workers are started with daemon=True, but multiprocessing only terminates daemonic children from the parent's atexit handler. When the parent is killed by SIGKILL, by SIGTERM with the default handler, or by the OOM killer, that handler never runs and the workers keep running and holding their memory. Each worker now starts a daemon thread that waits on the parent process sentinel. When the parent exits, the thread sets should_stop so the prefetch loop stops and cleans up its buffer as on a normal close, and exits the worker if it is still alive after 5 seconds, for example when it is blocked inside a user transform. Fixes google#1423.
mahdi-shafiei
added a commit
to avitai/datarax
that referenced
this pull request
Nov 19, 2026
test_a_training_process_dying_as_its_workers_start_leaves_none required every worker gone 2 s after its training process died. A worker still starting at that moment ends only once its set-up starts the watcher, after its spawn, imports and unpickling, and that time grows with load. On 24 CPUs it took 1.4-1.6 s at a load average near 12 and 1.7-3.3 s with 24 busy processes added, so [0] failed under load (a worker alive after 2 s) while every worker ended within 30 s. A worker now records when its watcher starts (TimedSetUp, beside UnwatchedSetUp), and each must be gone within 2 s of the later of the death and that start, and end within 30 s, the wait Grain's test of its own watcher uses (google/grain#1475). The 2 s bound is unchanged. A watcher planted 3 s late fails all four cases; the parent-watcher tests pass beside 24 busy processes.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1423. mp_prefetch workers are started with daemon=True, but multiprocessing only terminates daemonic children from the parent’s atexit handler, which does not run when the parent is killed by SIGKILL, by SIGTERM under the default handler, or by the OOM killer. The workers then keep running and holding memory.
Each worker now starts a daemon thread that waits on the parent process sentinel. It becomes ready when the parent exits, whichever thread started the worker, and works on Linux and macOS. When it fires, the thread sets should_stop so the prefetch loop stops and clears its buffer as on a normal close. If the worker is still alive after 5 seconds, for example because it is blocked inside a user transform, it calls os._exit. The forced-exit timer starts before should_stop is set, in case the parent died while holding the event’s lock. I did not use PR_SET_PDEATHSIG because it fires when the creating thread exits, and workers are spawned from prefetch threads that are joined on close and set_state.
The new test test_workers_exit_when_parent_is_killed runs an mp_prefetch pipeline in a subprocess, kills it with SIGKILL and with SIGTERM, and checks that both workers exit. It fails without the change. In my runs the workers exited about 0.6 s after the kill, and a worker blocked in a transform was stopped by the 5 s timeout. All of process_prefetch_test and prefetch_test pass.