Repository navigation
[turbopack] Fix GC/eviction race by dropping task resurrection - #100370
Merged
Merged
Conversation
GC marks tasks deleted under the snapshot exclusion, but eviction drops them after the tombstone commits without any exclusion. A racing get_or_create_task could recover a deleted task id from task_cache, eviction would then remove the task, and connect_child panicked in increase_active_count because the task no longer existed. GC now removes collected tasks from task_cache while operations are still excluded, and task_by_type skips deleted or missing candidates, so a reconnect mints a fresh task instead of reusing a dying id. That replaces resurrection entirely: GC already drops a collected task's edges and cell data, so reviving it in place bought nothing over a new task. With no legitimate way to reach a deleted task outside GC, ExecuteContext::task/task_or_create now debug-assert against it.
Contributor
Tests PassedCommit: 8151945 |
lukesandberg
marked this pull request as ready for review
October 19, 2026 21:33
…asks optimize_task and update_aggregation_number returned early on deleted tasks, but neither can reach one: GC's parallel collect runs no balance or optimize jobs, and its deferred rebalance starts from edges whose endpoints were filtered for deletion. Reaching a deleted task would mean edge removal left an edge behind, so assert instead.
jimmyhmiller
approved these changes
Nov 5, 2026
lukesandberg
added a commit
that referenced
this pull request
Nov 6, 2026
(cherry picked from commit 4a0609e) Co-authored-by: Luke Sandberg <lukesandberg@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
lukesandberg
added a commit
that referenced
this pull request
Nov 7, 2026
Backports these two Turbopack GC fixes to `releases/lts/active`: - [turbo-tasks-backend: keep recorded GC roots until their TTL expires (#100279)](#100279) - [[turbopack] Fix GC/eviction race by dropping task resurrection (#100370)](#100370) It builds on #100209 (the #99902 backport) and #100002 (snapshot consistency fixes), both already on this branch. GC is opt-in on 16.4 (`experimental.turbopackGc`), so this only affects apps that enable it. ## Conflict resolution **#100279** - `gc.rs` / `mod.rs` doc comments: took canary's text, which uses plain backticks instead of intra-doc links. - `snapshot_and_persist`: the `process` closure keeps this branch's 3-argument signature. #99747, which added a `category` parameter, isn't on this branch. The PR's `#[cfg(debug_assertions)] gc_protected_roots` set and the extended `debug_assert!` were added unchanged. **#100370** - `aggregation_update.rs` `increase_active_count`: replaced `resurrect_deleted(...)` with `task.assert_not_deleted(...)`. Kept the `self.check_optimization_pending(&task)` call that follows it on this branch; canary removed it in #100146/#100147, which aren't here. - `aggregation_update.rs` `optimize_queue.pop_front()` arm: kept the `optimization_pending_flag_already_set` binding. Canary renames it to `_`, but this branch still reads it in the per-job budget-exhausted path (`lock_and_mark_optimization_pending`). Canary replaced that path with a bulk drain. Everything else applied as-is. ## Verification - `cargo nextest run -p turbo-tasks-backend`: 289/289 passed. This includes the new `gc_cross_session` TTL tests and the `task_by_type_tests` lookup tests. - `cargo clippy -p turbo-tasks-backend --tests` is clean. - `cargo check -p turbo-tasks-backend` passes with `--release` and with `--features task_dirty_cause`. - `cargo fmt --check` is clean. CI may show the Tailwind failures currently affecting all `releases/lts/active` PRs until the Tailwind pin backport (#101307) lands. <!-- NEXT_JS_LLM --> <!-- fleet 91f3474f-6748-4c06-8329-d8d3673533dd --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
sokra
added a commit
that referenced
this pull request
Nov 10, 2026
…r debt instead of retrying (#100873) ### What? Replace aggregation-edge removal retries with **negative follower counts** ("debt"). A removal that overtakes its addition records debt; the late addition cancels it instead of spinning, requeuing, or eventually panicking. Remove `retry_loop`, the retry job fields, and `MAX_RETRIES`. Also add deterministic regression tests, concurrent graph-churn stress tests, fuzz-driver improvements, and benchmarks. ### How? An edge `U → F` lives either in `F.upper[U]` (unsigned) or in `U.followers[F]` (now `i32`). **Only followers can go negative. Only counts greater than zero are real edges.** - **Removal, without a pair lock:** decrement an existing inner edge under F's lock; otherwise decrement the follower count under U's lock, possibly creating debt. If debt remains, recheck F and schedule `BalanceEdge` if an inner edge appeared meanwhile. - **Addition:** pay existing debt on the follower side before creating an inner edge. After creating a new inner edge, recheck U for debt unconditionally, including empty-data/no-activeness cases. - **Race argument:** removal writes U then reads F; addition writes F then reads U, each under the relevant task lock. For a split that survives intervening operations, both reads cannot precede the other write; at least one side schedules balancing. An intervening update may already resolve the split. - **Pair-locked balancing:** net follower debt against the inner count first, reverse any disappeared inner contribution, then place the remaining count using root precedence and the existing equality-conflict rules. - **Semantic effects:** aggregate data, follower propagation, and active counts change only on `<= 0` ↔ `> 0` crossings. Debt creation/payment alone has no semantic edge effect. - **GC:** outgoing follower debt does **not** pin its owner. With persistence enabled, normal queues hold operation guards and snapshot/GC exclusion waits for operations to settle. GC finishes parallel deletion before deferred rebalancing. These phase guarantees—not a debt reference—protect in-flight updates. Removing the last raw debt entry still reports possible collectibility. - **Helpers:** `aggregation_edge.rs` contains pure checked arithmetic, placement, and netting. In the queue, the handlers for one or several uppers or followers share step helpers: removal `remove_inner_edges` → `on_inner_edges_disappeared` and `remove_follower_edges`; addition `follower_add_side` / `add_follower_edge` and `add_inner_edge` → `on_inner_edges_appeared` (`on_inner_edges_appeared_on_upper` for the multi-follower handler). Their id lists use a small `TaskIdList` trait (`TaskIdVec` or `Option<TaskId>`), so single-edge handlers build no vectors. - **Verifier:** at stop, scan restored resident metadata for residual debt, split edges, and invalid placement. Retain edgeless endpoints' numbers; do not fabricate numbers for unrestored metadata. Root follower edges are invalid even when the other endpoint is unavailable. ### Review follow-up A separate-model correctness review identified four issues, addressed by the follow-up: 1. Ignore intentional panic-catching/panic tests on aborting WASM targets, using the repository convention; native panic assertions still run. 2. Replace the helper-only addition race test with three **real-handler** tests. A per-queue, test-only one-shot hook forces removal to finish between the addition's precheck and inner write, without holding either task lock. Cover both empty and nonempty data. Independently removing each production postcheck callsite now makes its corresponding test fail. 3. Fix the verifier's false negative for resident, edgeless endpoints. Reinstating the old omission makes the new regression fail. Include valid follower placement and unknown/unrestored endpoint cases. 4. Correct the GC explanation and add actual collectibility and persistence-operation exclusion regressions. Keep GC semantics unchanged. Additional focused tests seed dirty/current-session-clean state and nonzero active counters: pure debt has no propagation effects; overpayment creates one appearance per edge; repeated positive references do not duplicate effects; split cancellation reverses dirty and clean contributions once; leaf promotion preserves data, active counts, and ancestor state while renesting dirty containers. **Review order:** pure helpers → queue module docs and handlers → snapshot/verifier → deterministic tests → churn/fuzz driver → benchmarks. ### Maintainer review round (0f405c3, 7b61001) - `0f405c3e`: plain `i32` arithmetic in `net_debt`; no `follower_ids` clone when there are no uppers; removal steps shared between single and multiple edges via `TaskIdList`; `check_optimization_pending` on the single upper; test hook type renamed to `TestHook`. - `7b61001b`: the three add handlers share step helpers like the removal path. Lock order, debt pre-check and postcheck, job variants, batching and optimize heuristics are unchanged. - Verified on the final state: full crate suite 56/56 groups in default, `verify_aggregation_graph` and `aggregation_update_no_batch`; churn 10/10 normal + 10/10 verify (serial); fuzz 10,449 runs / 181 s without crashes; clippy and rustdoc `-D warnings`. Removing either shared postcheck, or the debt pre-check in `follower_add_side`, fails the race/debt tests. CI green on `7b61001b`. - Queue microbench vs `c9d5d79f` (3 interleaved rounds, median): ordered −3%, ordered removals −10%, overtake −6%, **ordered additions +4–5%** (~6 ns per single-edge addition, attributed to the add-path extraction; not the added `check_optimization_pending` and not recovered by inline hints). ### Verification Follow-up local results: - Full crate suites in default, `verify_aggregation_graph`, and `aggregation_update_no_batch`: **56/56 test groups pass in each mode**, including **198 unit tests** (one ignored benchmark). - `aggregation_churn`: **20/20 serial runs in normal mode and 20/20 with graph verification**. - The known WASI failure on the original tip was deterministic (`catch_unwind` on panic-abort), not a flaky job. Target guards are fixed; **post-push WASI CI passes** on `e82c325d` (job `123330618672`). The full build-and-test suite is green, including native units, miri, Rust checks, benches, and Windows tests. - Four mutation probes: each of the three actual add-handler postcheck callsites, plus the verifier's edgeless-endpoint omission, is independently detected by the new tests; all mutations restored. - Follow-up fuzz rerun: **6,929 runs in 181 seconds**, no crashes, with graph verification enabled (no sanitizer). - Follow-up formatting, clippy (backend all targets, default and graph-verification modes; fuzz crate), and private rustdoc with `-D warnings`: pass. Historical pre-follow-up evidence (not newly measured coverage/performance): - Full crate suites in default, `verify_aggregation_graph`, and `aggregation_update_no_batch` modes passed. - Churn 20/20 in normal and graph-verification modes; nine existing stress targets 90/90. - Fuzz: 58,782 runs in 16 minutes, plus a later 14,736-run session, no crashes. - Coverage: helper file 100% of lines; changed queue lines excluding ignored benchmark 96.7%. These figures include in-file test code and **were not rerun for the review follow-up**. ### Benchmarks Historical same-machine A/B against canary, three interleaved rounds, criterion medians: | Workload | Change | |---|---| | Queue microbench, removals overtake additions | **−38%** | | Queue microbench, ordered add + remove (32 edges) | +3–4% (removals +14–20%, additions −2–8%) | | Concurrent end-to-end churn / `scope_stress` | within noise | | Ordered end-to-end churn | noisy, no stable direction | | `stress/fibonacci/500` | approximately +3–7% | This is a tradeoff, not a blanket performance win. The single-threaded microbench does not measure the old multi-threaded spin-wait. Part of the Fibonacci cost comes from the extra upper lock required by the race argument; reducing that cost without weakening correctness is a separate follow-up. No new performance claim is made for the review follow-up. ### Limitations / status - The verifier can only check placement using restored resident endpoint metadata; absent/evicted endpoint numbers remain unknown, except root placement can be checked without them. - **Unmatched debt is silent.** Debt is only valid if the matching addition is eventually generated and processed; producers must push additions into a queue their operation runs to completion. This is a requirement on producers, separate from the GC/snapshot exclusion rules (debt does not pin tasks). Production aggregation drains are synchronous (no async cancellation point) and now abort on panic rather than returning a partial graph to a per-task catcher. This does not protect jobs abandoned by a producer before the drain starts. If an addition were lost, a later addition of the same task pair would pay the stale debt instead of creating the edge, which can skew propagation. The old retry loop panicked in that case; now only the `verify_aggregation_graph` stop-time check reports residual debt (on restored resident tasks). - The stress test checks returned per-round counts. A stricter exact-collectibles check also fails on unmodified canary under CPU load and is tracked separately. - A second separate-model correctness review found no new blocking defect; its documentation note on the unmatched-debt assumption is applied above. Neither review is a formal proof or approval. ### Removal benchmark and traversal follow-up Two separate commits add permanent coverage and then the narrow traversal change: - `private::aggregation_queue` is an explicitly unstable native fixture shared by benches and normal-library profiling. The opt-in `TURBOPACK_BENCH_AGGREGATION_QUEUE=1` Criterion group runs ten routine cases; `extended` exposes 756 filterable configurations: inner/follower/mixed placement, true single-edge/one-ID batch/full-batch jobs, references 1→0 or 2→1, data/no-data, aligned/reversed/seeded-permuted IDs, and total sizes 1/8/16/32/33/64/66. Setup/restoration/checks are untimed. Raw edge counts and aggregate contributions are checked after every phase. Aggregation-number optimizations are deliberately disabled: this is isolated queue work, not an end-to-end workload. - `remove_follower_edges` consumes its input with existing `TaskIdList::retain` swap-removal semantics. For aligned small linear maps, visiting the swapped replacement avoids repeated scans. Both result lists, edge multiplicity, positive-crossing effects, lock order, debt prechecks and postchecks remain intact. Emitted batch order intentionally changes; new real-handler tests cover Option/32/33 inputs, ancestor propagation, retained references, duplicate IDs and mixed positive/debt entries. Normal-dependency profiling (not a backend test build) uses the same frozen fixture, system allocator, one runtime/backend worker, no backing storage, and no dev-only inline statistics. Release settings are explicit (opt3/thin LTO/cgu1 plus the applicable regex-syntax size override). Callgrind/DHAT cover 18 configurations; Criterion uses 30 samples, 1 s warmup, 3 s measurement and three interleaved parent/patch rounds. Payload/context/queue construction and removal drain are timed; restoration/assertions are not. Guest instructions and simulated events are not hardware cycles. | Synthetic removal case | Guest instructions | Median elapsed time | |---|---:|---:| | Aligned follower full batch, 32 edges | −9.79% | 1922.013 → 1743.011 ns (−9.31%) | | Follower full batch, 64 edges | +5.23% | 3440.070 → 3670.363 ns (+6.69%, first three rounds) | The 64-edge case had non-overlapping slower 95% intervals in all three rounds. Its extra confirmation pair was 3454.881 → 3569.930 ns (+3.33%), still with non-overlapping intervals: the ≥5% magnitude did not repeat, but a smaller slowdown persists. This tradeoff is retained and disclosed, not described as regression-free. Full33 is +4.78% elapsed/+5.99% instructions; mixed one-ID32 is +4.25% elapsed. Mixed/permuted/retained cases can do more instruction work. Allocation totals were equal across the three revisions in all 18 sampled cases. Remaining normal-library helper/guard/accessor overhead relative to canary is not fully diagnosed; symbol self costs do not prove extra semantic locks. No blanket speedup claim. Local verification before the canary rebase: all three full feature modes pass 56/56 groups (204 unit tests plus one ignored benchmark); 10/10 normal + 10/10 graph-verified serial churn; fuzz seed 2019458061 completed 14,271 runs in 181 s without failure; both shared addition-postcheck mutations fail the corresponding real-handler race tests and were restored. Formatting, all-target clippy, private rustdoc (warnings denied), fuzz manifest checks, 756-case extended smoke, ten routine cases, disabled gating and the original ignored benchmark all pass. Separate-model correctness and profiling-report reviews found no remaining blocker; these are not human approval. Post-push CI is green on `50e19c34`, including native cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc, and Windows tests; no reruns were needed. ### Canary rebase (`cc9ae702`) Rebased the seven existing commits onto canary `cc65c414`. The only manual integration resolves `TurboTasksBackend::stop`: keep canary's `ActivityMonitor` shutdown handling, then run both graph and aggregation-edge verification before dropping storage. All debt, race, GC, helper and traversal behavior is unchanged; the benchmark and traversal commits remain separate. Range-diff shows only this stop-hook adaptation. Both rebased layers were checked against exact verified source trees. On this parent, full default, `verify_aggregation_graph` and `aggregation_update_no_batch` suites each pass **59 test groups**, including **216 unit tests** (one ignored benchmark). Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest checks pass. The historical stress/fuzz/mutation and benchmark results above were **not rerun on the new canary base**, and their performance percentages are not new-base measurements. Post-push CI is green on `cc9ae702`, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests; no reruns were needed. ### GC/no-resurrection canary rebase (`c2d937c1`) Rebased the same seven commits onto canary `4a0609e1` (#100370). The only manual resolution removes the obsolete `resurrect_deleted` import while keeping aggregation-edge helpers. Upstream's live-task assertions, deleted-candidate filtering, GC task-cache removal and optimization changes remain intact. The previous `ActivityMonitor` stop-verification integration is retained. No debt, race, balancing or traversal logic was weakened; final source exactly matches the verified integration tree. Latest local verification: full default, `verify_aggregation_graph` and `aggregation_update_no_batch` suites pass **59 groups each**, including **220 unit tests** (one ignored benchmark), the updated GC reconnect/replacement tests and all four `task_by_type` tests. Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest check pass. On the child an existing 16-bit random-tag re-execution oracle collided once despite correct 1→2 propagation and logged re-execution; targeted and final full runs passed, with a deterministic-oracle follow-up recorded. No test assertions were removed. Older performance, fuzz and stress numbers above are historical, not measurements on this base. Post-push CI is green on `c2d937c1`, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests; no CI reruns were needed. ### Reviewer follow-up (`614290c4`, `75538d53`) - Added a narrow fatal-panic boundary to production aggregation queue drains, including stats and GC entry points. The manager's outer abort wrapper does not by itself protect backend calls reached inside the per-task panic catcher. Native subprocess tests first reproduced containment of an injected panic after a debt write, then verified process abort and normal controls. User-code panic policy is unchanged; synchronous drains have no async cancellation point, and pre-drain job abandonment remains a producer responsibility. - Shortened GC docs; renamed follower-edge propagation and activeness state; used `filter_map`; clarified the outgoing-diff merge TODO without implementing merging. - Extracted optimization heuristics without changing raw debt-inclusive counts, single power-of-two versus batch ceiling-power triggers, caps or pending flags. Added boundary/debt-count regressions. - Kept singleton peeling and Option/Vec job shapes. The shared iterator helper has no-storage and ready-task paths that do not allocate restore batches; no blanket allocation or speedup claim is made. Existing real-handler tests cover root/root placement and root debt payment. Local checks on both layers: three full feature modes pass **59 groups each** (parent **223** units, child **229**, one ignored benchmark); stats libraries, fmt, all-target clippy, private rustdoc with warnings denied, and fuzz manifest checks pass. Both layers pass opt-in benchmark smoke: **10 routine + 756 extended cases**, with default omission verified. These are correctness smoke tests, not fresh performance measurements. The historical timings/stress/fuzz results above were not rerun for this update. Pushed-tip CI is green on `75538d53`, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests; this parent's CI needed no reruns. Review threads remain unresolved for the reviewer. <!-- NEXT_JS_LLM --> <!-- fleet e31fbff5-6610-4d8e-9172-b2528e8725ad --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
sokra
added a commit
that referenced
this pull request
Nov 10, 2026
…s and balancing (#100892) Stacked on #100873 (negative follower debt). Only the commits on top of #100873 belong to this PR. ### What? Within one `AggregationUpdateQueue`, aggregation edge removals (`InnerOf{Upper,Uppers}LostFollower(s)`) now wait in a separate removal queue until all ordinary jobs, aggregation number updates and balancing in that queue are done. ### Why? Previously additions and removals were interleaved in one FIFO. When an edge (or a task reachable through another edge) was removed and re-added in the same queue, its aggregated data went `1 → 0 → 1`: data was removed and re-added, collectibles dependents were invalidated, dirty containers flipped and readers could briefly miss collectibles. With removals last it goes `1 → 2 → 1`, so none of that temporary work happens. ### How? - `push` routes the three lost-follower jobs into `removal_jobs`; everything else (data updates, active/parent counts, invalidations, additions) keeps its queue and order. - `process` priority: `jobs` → aggregation number updates → balancing → **removals** → optimize → find dirty → schedule. A small `pop_job` helper picks a removal only when the higher-priority queues are empty, so work generated by a removal runs before the next removal, and generated removals append FIFO. The existing job dispatch is unchanged. - GC: `execute_until_rebalance` must drain removals while balancing stays deferred. It moves balance jobs out of the queue after every step and returns both previously pending and newly generated ones; the caller's deleted-endpoint filtering is unchanged. - Safety: a removal processed later is the ordinary case. A removal that still overtakes its addition (another queue, or an addition generated later) records follower debt as in #100873. The PR relies on the existing debt contract and doesn't weaken it. No new locking, no schema change. ### Scope / limitations - Queue-local only: helps when the keeping addition/balance is already pending in the same queue. No atomicity across separate queues or concurrent operations. - Task completion still removes old children (`cleanup_old_edges`) and connects new children (`connect_children`) in separate queues, so a task switching between children that share a descendant still churns. Combining those is planned as a separate, later stacked PR after measuring this one. ### Tests New `deferred_removal_tests` (real backend, step-by-step queue processing): - same-edge replacement and a moved shared descendant: the upper's collectible never drops to zero at any step, and a collectibles reader is not invalidated; - genuine loss still removes data and invalidates the reader; - removals only run when no ordinary job, aggregation number update or balance is pending; - work generated by a removal runs before the next removal, generated removals append FIFO; - `execute_until_rebalance` drains removals but executes no balancing and returns pending and newly generated balance edges. An existing debt test that mixed remove/remove/add in one queue now uses separate queues so it still forces a real overtake. Mutations: routing removals back into the ordinary FIFO fails 4 of the new tests; not collecting newly generated balances in the GC path fails the GC test. ### Verification - `cargo test -p turbo-tasks-backend` in default, `verify_aggregation_graph`, and `aggregation_update_no_batch`: 56/56 test groups each (including the existing stress tests). - `aggregation_churn` stress: 20/20 serial runs normal and 20/20 with graph verification. - `fuzz_graph`: 8,543 runs in 181 s, no crashes. - clippy (`--all-targets`, default and `verify_aggregation_graph,trace_aggregation_update_stats`), fuzz crate clippy, fmt, private rustdoc with `-D warnings`. ### Work counts and performance Exact counts (`replace_work`: 32 existing edges removed and re-added in one queue, a reader on the upper; feature `trace_aggregation_update_stats`), vs #100873: | | aggregated data update jobs | collectibles invalidation jobs | |---|---:|---:| | #100873 | 2 | 2 | | this PR | 0 | 0 | Queue microbench (release unit-test binary, 3 interleaved same-machine rounds, Criterion central estimates): | case | #100873 | this PR | |---|---|---| | replace (remove + re-add in one queue) | 12.7–12.9 µs | 8.5–10.2 µs (−20…−35%) | | overtake (removals in a separate earlier queue) | 10.2–10.4 µs | 11.0–12.3 µs (+6…+21%, noisy) | | ordered / ordered additions / ordered removals | — | within noise, removals possibly slightly slower | The microbench binary includes inert test-only hooks on this branch, so it does not isolate production overhead. End-to-end benches (`aggregation_churn`, `stress/fibonacci/500`) drifted too much across rounds to draw a conclusion either way. This is not claimed as a general speedup. ### Rebase onto removal benchmark/traversal follow-up Rebased the existing two logical commits onto #100873 at `50e19c34`, retaining the deferred-removal change and separate balance-queue drain refinement. The parent's permanent removal fixture/benchmarks and swap-order traversal are inherited, not additional child scope. Only the new parent's test observer needed adapting to inspect `removal_jobs` rather than the ordinary queue; production priority, GC balance collection and drain behavior are unchanged by the rebase. Local verification before the canary rebase: full default, `verify_aggregation_graph` and `aggregation_update_no_batch` suites each pass **56/56 groups** (210 unit tests plus one ignored benchmark). This includes the parent's 756-configuration fixture test and traversal/multiplicity/ancestor regressions, the actual-handler race tests, and this child's priority, genuine-loss, no-invalidation and GC-drain tests. Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest check pass. The older stress/fuzz/performance numbers above are historical, not newly measured on this rebased child. Parent normal-library profiling explicitly reports a 32-edge benefit and a larger-map tradeoff; it is not evidence of an end-to-end child speedup. Post-push CI is green on `c26cd62d`, including native cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc, and Windows tests; no reruns were needed. ### Canary restack (`e9607690`) Replayed the existing two child commits onto #100873 at `cc9ae702` (canary base `cc65c414`). Both child patches are unchanged by range-diff. Queue priority, GC balance collection and separate drain refinement are preserved. The final source tree exactly matches the locally verified integration tree. On this child, full default, `verify_aggregation_graph` and `aggregation_update_no_batch` suites each pass **59 test groups**, including **222 unit tests** (one ignored benchmark). Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest checks pass. Prior stress/fuzz/performance results above remain historical; no new-base performance or end-to-end speedup is claimed. Post-push CI is green on `e9607690`, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests; no reruns were needed. Base relationship and ready status are unchanged; higher layers were not modified. ### GC/no-resurrection canary restack (`d91b5264`) Restacked the same two child commits onto #100873 at `c2d937c1` (canary base `4a0609e1`, #100370). Both child patches remain unchanged by range-diff: removal priority, deferred GC balance collection and the separate drain refinement are preserved. Upstream no-resurrection behavior and deleted-endpoint filtering are retained. The final source exactly matches the verified integration tree. Latest local verification: full default, `verify_aggregation_graph` and `aggregation_update_no_batch` suites pass **59 groups each**, including **226 unit tests** (one ignored benchmark), updated GC reconnect/replacement and `task_by_type` tests, plus this child's GC/priority regressions. Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest check pass. Initial default `recompute_dependency` hit an existing 16-bit random-tag collision: both functions logged re-execution and value correctly changed 1→2, but tags were equal. A targeted rerun and final full runs pass; a deterministic-oracle follow-up is recorded, without changing test or production behavior here. Older stress/fuzz/performance figures remain historical; no new-base speedup is claimed. Post-push CI is green on `d91b5264`, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests; no CI reruns were needed. Base, ready status and higher stack layers are unchanged by this update. ### Restack onto reviewer follow-up (`fd4345e9`) Replayed only the existing two logical child commits onto #100873 at `75538d53`. The GC drain keeps its removal-job predicate, deferred balance collection and per-step `drain(..)` inside the parent's new fatal-panic boundary. Final source matches the verified child integration exactly. Queue priority, debt pre/postchecks, deleted-endpoint filtering, PR base/readiness and higher stack branches are unchanged. Local verification: default, `verify_aggregation_graph` and `aggregation_update_no_batch` full suites each pass **59 groups**, including **229 unit tests** (one ignored benchmark); stats library passes (two ignored benchmarks), including ordinary/stats/GC fatal-panic subprocess tests. Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest checks pass. Opt-in queue benchmark smoke passes **10 routine + 756 extended cases**, and default omission is checked. Older timing/stress/fuzz figures above remain historical; no new speedup is claimed. Pushed-tip CI is green on `fd4345e9`, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests. The first cargo-unit run had one `turbopack-go::live::versioned_dependencies_use_the_module_cache_without_downloads` failure with `Permission denied (os error 13)`; one rerun passed without source changes. That test's error-context/permission investigation is tracked separately. <!-- NEXT_JS_LLM --> <!-- fleet e31fbff5-6610-4d8e-9172-b2528e8725ad --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix a panic in development:
connect_child→increase_active_counthit a task that GC had collected and eviction had already removed (task is missing in memory or persistent storage).The race. GC marks tasks deleted under the snapshot exclusion. The exclusion is released before the snapshot commits the tombstones, and eviction then removes the deleted tasks with no exclusion. A concurrent
get_or_create_taskcould read a deleted task's id from the in memorytask_cache, eviction could remove the task, and the connect would then panic. Resurrection only worked if the connect finished before eviction.The fix. A deleted task's id is never handed out again from the disk cache:
task_cachewhile operations are still excluded.task_by_typeskips candidates that are deleted or already gone.task_cachefor deleted tasks. (it doesn't need to, there are no entries)Resurrection is removed. The only thing resurrection is really saving is allocating a new
TaskId(cells and edges are already gone), so it is simpler to remove this path.Since nothing outside GC can legitimately reach a deleted task now,
ExecuteContext::taskandtask_or_createdebug-assert against one, except when we are running GC, since GC marks tasks deleted and then removes their edges.gc_resurrection.rstests are renamed and re-documented for the fresh-task behavior; their assertions are unchanged. Addedgc_reconnect_persisted_before_tombstone_commits, which reconnects a persisted, collected subtree while the type index still names the old tasks. That test covers the new lookup path; it does not reproduce the original timing race, which I couldn't trigger deterministically.Verification
cargo nextest r -p turbo-tasks-backend(295/295)cargo clippy -p turbo-tasks-backend --tests, pluscargo check -p turbo-tasks-backend --features task_dirty_cause