Repository navigation
systrap: fail-stop sentry on stuck context - #14531
Merged
Merged
Conversation
A systrap task could wait in sleepOnState forever when its stub stopped responding: one interrupt was sent, then the loop only logged warnings. Container teardown then blocked indefinitely in WaitExited, wedging the sandbox and its control RPCs (pods stuck Terminating for days, see #14408 and #14405). A single interrupt can be lost (#14405), so resend it on every checkup wakeup. If the stub stays unresponsive through the 30s stuck deadline, or is already gone, dump all goroutine stacks and kill only the stuck subprocess through the path NotifyInterrupt already uses (mark dead, ContextStateUnexpectedDeath, kill the syscall thread). The task goroutine returns instead of blocking Kernel.Pause() forever, and healthy subprocesses are unaffected. Fixes #14408 Assisted-by: Claude Code
copybara-service
Bot
force-pushed
the
test/cl974017442
branch
2 times, most recently
from
August 31, 2026 23:05
3558371 to
bf483bd
Compare
PiperOrigin-RevId: 974146030
copybara-service
Bot
force-pushed
the
test/cl974017442
branch
from
August 31, 2026 23:47
bf483bd to
b0adda2
Compare
copybara-service Bot
pushed a commit
that referenced
this pull request
Sep 2, 2026
github.com//pull/14531 exposed a race window with stub threads setting ThreadID and the sentry reading it. If the sentry attempted to interrupt a stub that in the process of switching a context back, it could have read an invalid thread ID along with a context state that indicated it was still being worked on. This window has existed before #14531, but was relatively benign; with that CL, the window became punishable by subprocess death. The (mostly generated) test seems to reliably test the race-window in case things get reordered in some future refactor. Fixes #14531. PiperOrigin-RevId: 974834558
copybara-service Bot
pushed a commit
that referenced
this pull request
Sep 2, 2026
github.com//pull/14531 exposed a race window with stub threads setting ThreadID and the sentry reading it. If the sentry attempted to interrupt a stub that in the process of switching a context back, it could have read an invalid thread ID along with a context state that indicated it was still being worked on. This window has existed before #14531, but was relatively benign; with that CL, the window became punishable by subprocess death. The (mostly generated) test seems to reliably test the race-window in case things get reordered in some future refactor. Fixes #14531. PiperOrigin-RevId: 974868738
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
systrap: fail-stop sentry on stuck context
A systrap task can remain in
sleepOnStateforever when its stub stops responding. The existing path sends a single interrupt and logs repeated warnings, but it has no retry and no escape. Container teardown can then block indefinitely inWaitExited, leaving the sandbox and its control RPCs stuck.We hit this in a production Kubernetes cluster. Multiple gVisor pods remained Terminating for days, on both release-20260427.0 and release-20260714.0. In one affected sandbox, both
runsc killandrunsc debug --stackshung while the sentry stayed alive. Systrap[exe]stubs and node load continued to accumulate even though CPU use remained low. Killing the sentry process tree cleared the pod and returned node load to normal immediately. Full forensics in #14408.#14405 is an independent production report of the same hang, with the sentry stacks we could not capture: a task goroutine parked 7,818 minutes in
sleepOnStateon the ThreadContext.State futex after one missed SIGUSR1,Kernel.Pause()blocked behind it so the URPC handler forrunsc kill --allnever returns. On the shim side 45,646 goroutines piled up onInit.mubehind the hung kill, growing containerd and kubelet memory until node OOM.The fix, staged from least to most disruptive:
log.TracebackAll. Stacks are unobtainable from outside a wedged sandbox, so the escalation self-documents what systrap: sleepOnState() never returns when a stub thread is unresponsive, permanently hangs Kernel.Pause() #14405 had to capture by hand.NotifyInterruptalready uses when a stub is gone (mark dead,ContextStateUnexpectedDeath, kill the syscall thread). Healthy subprocesses in the sandbox are unaffected, and the task goroutine returns instead of blockingKernel.Pause()forever.sleepOnStateWithTimeoutreturns a typed error and stays side-effect free;sleepOnStatedoes the traceback and the kill. A stub that is already gone takes the quietNotifyInterruptpath (warn + kill subprocess, no stack dump). Resent interrupts do not extend the stuck deadline; a regression test covers that.Tests:
bazel test //pkg/sentry/platform/systrap:systrap_testbazel build //runsc:runscFixes #14408. Related to #14405 and #12209.
Assisted-by: Claude Code
FUTURE_COPYBARA_INTEGRATE_REVIEW=#14201 from NahumLitvin:fix/systrap-stuck-context d119f5c