Skip to content

systrap: fail-stop sentry on stuck context - #14531

Merged
copybara-service[bot] merged 2 commits into
masterfrom
test/cl974017442
Aug 31, 2026
Merged

copybara-service[bot] merged 2 commits into
masterfrom
test/cl974017442

Conversation

@copybara-service

Copy link
Copy Markdown

systrap: fail-stop sentry on stuck context

A systrap task can remain in sleepOnState forever when its stub stops responding. The existing path sends a single interrupt and logs repeated warnings, but it has no retry and no escape. Container teardown can then block indefinitely in WaitExited, leaving the sandbox and its control RPCs stuck.

We hit this in a production Kubernetes cluster. Multiple gVisor pods remained Terminating for days, on both release-20260427.0 and release-20260714.0. In one affected sandbox, both runsc kill and runsc debug --stacks hung while the sentry stayed alive. Systrap [exe] stubs and node load continued to accumulate even though CPU use remained low. Killing the sentry process tree cleared the pod and returned node load to normal immediately. Full forensics in #14408.

#14405 is an independent production report of the same hang, with the sentry stacks we could not capture: a task goroutine parked 7,818 minutes in sleepOnState on the ThreadContext.State futex after one missed SIGUSR1, Kernel.Pause() blocked behind it so the URPC handler for runsc kill --all never returns. On the shim side 45,646 goroutines piled up on Init.mu behind the hung kill, growing containerd and kubelet memory until node OOM.

The fix, staged from least to most disruptive:

  1. Because systrap: sleepOnState() never returns when a stub thread is unresponsive, permanently hangs Kernel.Pause() #14405 shows a single interrupt can simply be lost, the interrupt is resent on every checkup wakeup instead of once. A lost signal recovers the workload with no visible damage.
  2. If the stub stays unresponsive through repeated interrupts for the full stuck-context deadline (30 seconds), all goroutine stacks are dumped via log.TracebackAll. Stacks are unobtainable from outside a wedged sandbox, so the escalation self-documents what systrap: sleepOnState() never returns when a stub thread is unresponsive, permanently hangs Kernel.Pause() #14405 had to capture by hand.
  3. Only the stuck subprocess is then killed, through the same path NotifyInterrupt already uses when a stub is gone (mark dead, ContextStateUnexpectedDeath, kill the syscall thread). Healthy subprocesses in the sandbox are unaffected, and the task goroutine returns instead of blocking Kernel.Pause() forever.

sleepOnStateWithTimeout returns a typed error and stays side-effect free; sleepOnState does the traceback and the kill. A stub that is already gone takes the quiet NotifyInterrupt path (warn + kill subprocess, no stack dump). Resent interrupts do not extend the stuck deadline; a regression test covers that.

Tests:

  • bazel test //pkg/sentry/platform/systrap:systrap_test
  • bazel build //runsc:runsc
  • Stuck, stub-gone, repeated-interrupt, and recovered-context cases covered on the typed error value.

Fixes #14408. Related to #14405 and #12209.

Assisted-by: Claude Code
FUTURE_COPYBARA_INTEGRATE_REVIEW=#14201 from NahumLitvin:fix/systrap-stuck-context d119f5c

A systrap task could wait in sleepOnState forever when its stub stopped
responding: one interrupt was sent, then the loop only logged warnings.
Container teardown then blocked indefinitely in WaitExited, wedging the
sandbox and its control RPCs (pods stuck Terminating for days, see #14408
and #14405).

A single interrupt can be lost (#14405), so resend it on every checkup
wakeup. If the stub stays unresponsive through the 30s stuck deadline, or
is already gone, dump all goroutine stacks and kill only the stuck
subprocess through the path NotifyInterrupt already uses (mark dead,
ContextStateUnexpectedDeath, kill the syscall thread). The task goroutine
returns instead of blocking Kernel.Pause() forever, and healthy
subprocesses are unaffected.

Fixes #14408

Assisted-by: Claude Code
@copybara-service copybara-service Bot added the exported Issue was exported automatically label Aug 31, 2026
@copybara-service
copybara-service Bot force-pushed the test/cl974017442 branch 2 times, most recently from 3558371 to bf483bd Compare August 31, 2026 23:05
@copybara-service
copybara-service Bot merged commit b0adda2 into master Aug 31, 2026
5 of 8 checks passed
@copybara-service
copybara-service Bot deleted the test/cl974017442 branch August 31, 2026 23:48
copybara-service Bot pushed a commit that referenced this pull request Sep 2, 2026
github.com//pull/14531 exposed a race window with stub threads
setting ThreadID and the sentry reading it. If the sentry attempted to interrupt
a stub that in the process of switching a context back, it could have read
an invalid thread ID along with a context state that indicated it was still
being worked on. This window has existed before #14531, but was relatively
benign; with that CL, the window became punishable by subprocess death.

The (mostly generated) test seems to reliably test the race-window in case
things get reordered in some future refactor.

Fixes #14531.

PiperOrigin-RevId: 974834558
copybara-service Bot pushed a commit that referenced this pull request Sep 2, 2026
github.com//pull/14531 exposed a race window with stub threads
setting ThreadID and the sentry reading it. If the sentry attempted to interrupt
a stub that in the process of switching a context back, it could have read
an invalid thread ID along with a context state that indicated it was still
being worked on. This window has existed before #14531, but was relatively
benign; with that CL, the window became punishable by subprocess death.

The (mostly generated) test seems to reliably test the race-window in case
things get reordered in some future refactor.

Fixes #14531.

PiperOrigin-RevId: 974868738
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exported Issue was exported automatically

Projects

None yet

Development

Successfully merging this pull request may close these issues.

systrap: sentry unresponsive with stuck contexts; pod wedged in Terminating, only SIGKILL of sentry tree recovers

2 participants

⌂ 换网址