Skip to content

Stop mp_prefetch workers when the parent process dies - #1475

Open
princemittalr wants to merge 1 commit into
google:mainfrom
princemittalr:fix/mp-prefetch-orphan-workers
Open

princemittalr wants to merge 1 commit into
google:mainfrom
princemittalr:fix/mp-prefetch-orphan-workers

Conversation

@princemittalr

Copy link
Copy Markdown

Fixes #1423. mp_prefetch workers are started with daemon=True, but multiprocessing only terminates daemonic children from the parent’s atexit handler, which does not run when the parent is killed by SIGKILL, by SIGTERM under the default handler, or by the OOM killer. The workers then keep running and holding memory.

Each worker now starts a daemon thread that waits on the parent process sentinel. It becomes ready when the parent exits, whichever thread started the worker, and works on Linux and macOS. When it fires, the thread sets should_stop so the prefetch loop stops and clears its buffer as on a normal close. If the worker is still alive after 5 seconds, for example because it is blocked inside a user transform, it calls os._exit. The forced-exit timer starts before should_stop is set, in case the parent died while holding the event’s lock. I did not use PR_SET_PDEATHSIG because it fires when the creating thread exits, and workers are spawned from prefetch threads that are joined on close and set_state.

The new test test_workers_exit_when_parent_is_killed runs an mp_prefetch pipeline in a subprocess, kills it with SIGKILL and with SIGTERM, and checks that both workers exit. It fails without the change. In my runs the workers exited about 0.6 s after the kill, and a worker blocked in a transform was stopped by the 5 s timeout. All of process_prefetch_test and prefetch_test pass.

Prefetch workers are started with daemon=True, but multiprocessing only
terminates daemonic children from the parent's atexit handler. When the
parent is killed by SIGKILL, by SIGTERM with the default handler, or by
the OOM killer, that handler never runs and the workers keep running and
holding their memory.

Each worker now starts a daemon thread that waits on the parent process
sentinel. When the parent exits, the thread sets should_stop so the
prefetch loop stops and cleans up its buffer as on a normal close, and
exits the worker if it is still alive after 5 seconds, for example when
it is blocked inside a user transform.

Fixes google#1423.
mahdi-shafiei added a commit to avitai/datarax that referenced this pull request Nov 19, 2026
test_a_training_process_dying_as_its_workers_start_leaves_none required
every worker gone 2 s after its training process died. A worker still
starting at that moment ends only once its set-up starts the watcher,
after its spawn, imports and unpickling, and that time grows with load.
On 24 CPUs it took 1.4-1.6 s at a load average near 12 and 1.7-3.3 s
with 24 busy processes added, so [0] failed under load (a worker alive
after 2 s) while every worker ended within 30 s.

A worker now records when its watcher starts (TimedSetUp, beside
UnwatchedSetUp), and each must be gone within 2 s of the later of the
death and that start, and end within 30 s, the wait Grain's test of its
own watcher uses (google/grain#1475). The 2 s bound is unchanged. A
watcher planted 3 s late fails all four cases; the parent-watcher tests
pass beside 24 busy processes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mp_prefetch worker processes keep running after the main process is killed (SIGKILL/SIGTERM)

1 participant

⌂ 换网址