Skip to content

[turbopack] Fix GC/eviction race by dropping task resurrection - #100370

Merged
lukesandberg merged 3 commits into
canaryfrom
eviction_race
Nov 5, 2026
Merged

lukesandberg merged 3 commits into
canaryfrom
eviction_race

Conversation

@lukesandberg

@lukesandberg lukesandberg commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fix a panic in development: connect_child → increase_active_count hit a task that GC had collected and eviction had already removed (task is missing in memory or persistent storage).

The race. GC marks tasks deleted under the snapshot exclusion. The exclusion is released before the snapshot commits the tombstones, and eviction then removes the deleted tasks with no exclusion. A concurrent get_or_create_task could read a deleted task's id from the in memory task_cache, eviction could remove the task, and the connect would then panic. Resurrection only worked if the connect finished before eviction.

The fix. A deleted task's id is never handed out again from the disk cache:

  • GC removes collected tasks from task_cache while operations are still excluded.
  • task_by_type skips candidates that are deleted or already gone.
    • This is only possible for calls that race with snapshotting. We treat deleted tasks as absent (in other words like the commit already happened).
  • Eviction no longer touches task_cache for deleted tasks. (it doesn't need to, there are no entries)

Resurrection is removed. The only thing resurrection is really saving is allocating a new TaskId (cells and edges are already gone), so it is simpler to remove this path.

Since nothing outside GC can legitimately reach a deleted task now, ExecuteContext::task and task_or_create debug-assert against one, except when we are running GC, since GC marks tasks deleted and then removes their edges.

gc_resurrection.rs tests are renamed and re-documented for the fresh-task behavior; their assertions are unchanged. Added gc_reconnect_persisted_before_tombstone_commits, which reconnects a persisted, collected subtree while the type index still names the old tasks. That test covers the new lookup path; it does not reproduce the original timing race, which I couldn't trigger deterministically.

Verification

  • cargo nextest r -p turbo-tasks-backend (295/295)
  • cargo clippy -p turbo-tasks-backend --tests, plus cargo check -p turbo-tasks-backend --features task_dirty_cause

GC marks tasks deleted under the snapshot exclusion, but eviction drops
them after the tombstone commits without any exclusion. A racing
get_or_create_task could recover a deleted task id from task_cache,
eviction would then remove the task, and connect_child panicked in
increase_active_count because the task no longer existed.

GC now removes collected tasks from task_cache while operations are
still excluded, and task_by_type skips deleted or missing candidates, so
a reconnect mints a fresh task instead of reusing a dying id. That
replaces resurrection entirely: GC already drops a collected task's
edges and cell data, so reviving it in place bought nothing over a new
task. With no legitimate way to reach a deleted task outside GC,
ExecuteContext::task/task_or_create now debug-assert against it.
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Tests Passed

Commit: 8151945

@github-actions github-actions Bot added created-by: Turbopack team PRs by the Turbopack team. Turbopack Related to Turbopack with Next.js. labels Oct 18, 2026
@lukesandberg
lukesandberg requested a review from a team October 19, 2026 21:33
@lukesandberg
lukesandberg marked this pull request as ready for review October 19, 2026 21:33
…asks

optimize_task and update_aggregation_number returned early on deleted
tasks, but neither can reach one: GC's parallel collect runs no balance
or optimize jobs, and its deferred rebalance starts from edges whose
endpoints were filtered for deletion. Reaching a deleted task would mean
edge removal left an edge behind, so assert instead.
@lukesandberg
lukesandberg merged commit 4a0609e into canary Nov 5, 2026
102 checks passed
@lukesandberg
lukesandberg deleted the eviction_race branch November 5, 2026 21:33
lukesandberg added a commit that referenced this pull request Nov 6, 2026
(cherry picked from commit 4a0609e)

Co-authored-by: Luke Sandberg <lukesandberg@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
lukesandberg added a commit that referenced this pull request Nov 7, 2026
Backports these two Turbopack GC fixes to `releases/lts/active`:
- [turbo-tasks-backend: keep recorded GC roots until their TTL expires
(#100279)](#100279)
- [[turbopack] Fix GC/eviction race by dropping task resurrection
(#100370)](#100370)

It builds on #100209 (the #99902 backport) and #100002 (snapshot
consistency fixes), both already on this branch. GC is opt-in on 16.4
(`experimental.turbopackGc`), so this only affects apps that enable it.

## Conflict resolution

**#100279**
- `gc.rs` / `mod.rs` doc comments: took canary's text, which uses plain
backticks instead of intra-doc links.
- `snapshot_and_persist`: the `process` closure keeps this branch's
3-argument signature. #99747, which added a `category` parameter, isn't
on this branch. The PR's `#[cfg(debug_assertions)] gc_protected_roots`
set and the extended `debug_assert!` were added unchanged.

**#100370**
- `aggregation_update.rs` `increase_active_count`: replaced
`resurrect_deleted(...)` with `task.assert_not_deleted(...)`. Kept the
`self.check_optimization_pending(&task)` call that follows it on this
branch; canary removed it in #100146/#100147, which aren't here.
- `aggregation_update.rs` `optimize_queue.pop_front()` arm: kept the
`optimization_pending_flag_already_set` binding. Canary renames it to
`_`, but this branch still reads it in the per-job budget-exhausted path
(`lock_and_mark_optimization_pending`). Canary replaced that path with a
bulk drain.

Everything else applied as-is.

## Verification

- `cargo nextest run -p turbo-tasks-backend`: 289/289 passed. This
includes the new `gc_cross_session` TTL tests and the
`task_by_type_tests` lookup tests.
- `cargo clippy -p turbo-tasks-backend --tests` is clean.
- `cargo check -p turbo-tasks-backend` passes with `--release` and with
`--features task_dirty_cause`.
- `cargo fmt --check` is clean.

CI may show the Tailwind failures currently affecting all
`releases/lts/active` PRs until the Tailwind pin backport (#101307)
lands.

<!-- NEXT_JS_LLM -->


<!-- fleet 91f3474f-6748-4c06-8329-d8d3673533dd -->

---------

Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
sokra added a commit that referenced this pull request Nov 10, 2026
…r debt instead of retrying (#100873)

### What?

Replace aggregation-edge removal retries with **negative follower
counts** ("debt"). A removal that overtakes its addition records debt;
the late addition cancels it instead of spinning, requeuing, or
eventually panicking. Remove `retry_loop`, the retry job fields, and
`MAX_RETRIES`.

Also add deterministic regression tests, concurrent graph-churn stress
tests, fuzz-driver improvements, and benchmarks.

### How?

An edge `U → F` lives either in `F.upper[U]` (unsigned) or in
`U.followers[F]` (now `i32`). **Only followers can go negative. Only
counts greater than zero are real edges.**

- **Removal, without a pair lock:** decrement an existing inner edge
under F's lock; otherwise decrement the follower count under U's lock,
possibly creating debt. If debt remains, recheck F and schedule
`BalanceEdge` if an inner edge appeared meanwhile.
- **Addition:** pay existing debt on the follower side before creating
an inner edge. After creating a new inner edge, recheck U for debt
unconditionally, including empty-data/no-activeness cases.
- **Race argument:** removal writes U then reads F; addition writes F
then reads U, each under the relevant task lock. For a split that
survives intervening operations, both reads cannot precede the other
write; at least one side schedules balancing. An intervening update may
already resolve the split.
- **Pair-locked balancing:** net follower debt against the inner count
first, reverse any disappeared inner contribution, then place the
remaining count using root precedence and the existing equality-conflict
rules.
- **Semantic effects:** aggregate data, follower propagation, and active
counts change only on `<= 0` ↔ `> 0` crossings. Debt creation/payment
alone has no semantic edge effect.
- **GC:** outgoing follower debt does **not** pin its owner. With
persistence enabled, normal queues hold operation guards and snapshot/GC
exclusion waits for operations to settle. GC finishes parallel deletion
before deferred rebalancing. These phase guarantees—not a debt
reference—protect in-flight updates. Removing the last raw debt entry
still reports possible collectibility.
- **Helpers:** `aggregation_edge.rs` contains pure checked arithmetic,
placement, and netting. In the queue, the handlers for one or several
uppers or followers share step helpers: removal `remove_inner_edges` →
`on_inner_edges_disappeared` and `remove_follower_edges`; addition
`follower_add_side` / `add_follower_edge` and `add_inner_edge` →
`on_inner_edges_appeared` (`on_inner_edges_appeared_on_upper` for the
multi-follower handler). Their id lists use a small `TaskIdList` trait
(`TaskIdVec` or `Option<TaskId>`), so single-edge handlers build no
vectors.
- **Verifier:** at stop, scan restored resident metadata for residual
debt, split edges, and invalid placement. Retain edgeless endpoints'
numbers; do not fabricate numbers for unrestored metadata. Root follower
edges are invalid even when the other endpoint is unavailable.

### Review follow-up

A separate-model correctness review identified four issues, addressed by
the follow-up:

1. Ignore intentional panic-catching/panic tests on aborting WASM
targets, using the repository convention; native panic assertions still
run.
2. Replace the helper-only addition race test with three
**real-handler** tests. A per-queue, test-only one-shot hook forces
removal to finish between the addition's precheck and inner write,
without holding either task lock. Cover both empty and nonempty data.
Independently removing each production postcheck callsite now makes its
corresponding test fail.
3. Fix the verifier's false negative for resident, edgeless endpoints.
Reinstating the old omission makes the new regression fail. Include
valid follower placement and unknown/unrestored endpoint cases.
4. Correct the GC explanation and add actual collectibility and
persistence-operation exclusion regressions. Keep GC semantics
unchanged.

Additional focused tests seed dirty/current-session-clean state and
nonzero active counters: pure debt has no propagation effects;
overpayment creates one appearance per edge; repeated positive
references do not duplicate effects; split cancellation reverses dirty
and clean contributions once; leaf promotion preserves data, active
counts, and ancestor state while renesting dirty containers.

**Review order:** pure helpers → queue module docs and handlers →
snapshot/verifier → deterministic tests → churn/fuzz driver →
benchmarks.

### Maintainer review round (0f405c3, 7b61001)

- `0f405c3e`: plain `i32` arithmetic in `net_debt`; no `follower_ids`
clone when there are no uppers; removal steps shared between single and
multiple edges via `TaskIdList`; `check_optimization_pending` on the
single upper; test hook type renamed to `TestHook`.
- `7b61001b`: the three add handlers share step helpers like the removal
path. Lock order, debt pre-check and postcheck, job variants, batching
and optimize heuristics are unchanged.
- Verified on the final state: full crate suite 56/56 groups in default,
`verify_aggregation_graph` and `aggregation_update_no_batch`; churn
10/10 normal + 10/10 verify (serial); fuzz 10,449 runs / 181 s without
crashes; clippy and rustdoc `-D warnings`. Removing either shared
postcheck, or the debt pre-check in `follower_add_side`, fails the
race/debt tests. CI green on `7b61001b`.
- Queue microbench vs `c9d5d79f` (3 interleaved rounds, median): ordered
−3%, ordered removals −10%, overtake −6%, **ordered additions +4–5%**
(~6 ns per single-edge addition, attributed to the add-path extraction;
not the added `check_optimization_pending` and not recovered by inline
hints).

### Verification

Follow-up local results:

- Full crate suites in default, `verify_aggregation_graph`, and
`aggregation_update_no_batch`: **56/56 test groups pass in each mode**,
including **198 unit tests** (one ignored benchmark).
- `aggregation_churn`: **20/20 serial runs in normal mode and 20/20 with
graph verification**.
- The known WASI failure on the original tip was deterministic
(`catch_unwind` on panic-abort), not a flaky job. Target guards are
fixed; **post-push WASI CI passes** on `e82c325d` (job `123330618672`).
The full build-and-test suite is green, including native units, miri,
Rust checks, benches, and Windows tests.

- Four mutation probes: each of the three actual add-handler postcheck
callsites, plus the verifier's edgeless-endpoint omission, is
independently detected by the new tests; all mutations restored.
- Follow-up fuzz rerun: **6,929 runs in 181 seconds**, no crashes, with
graph verification enabled (no sanitizer).
- Follow-up formatting, clippy (backend all targets, default and
graph-verification modes; fuzz crate), and private rustdoc with `-D
warnings`: pass.

Historical pre-follow-up evidence (not newly measured
coverage/performance):
- Full crate suites in default, `verify_aggregation_graph`, and
`aggregation_update_no_batch` modes passed.
- Churn 20/20 in normal and graph-verification modes; nine existing
stress targets 90/90.
- Fuzz: 58,782 runs in 16 minutes, plus a later 14,736-run session, no
crashes.
- Coverage: helper file 100% of lines; changed queue lines excluding
ignored benchmark 96.7%. These figures include in-file test code and
**were not rerun for the review follow-up**.

### Benchmarks

Historical same-machine A/B against canary, three interleaved rounds,
criterion medians:

| Workload | Change |
|---|---|
| Queue microbench, removals overtake additions | **−38%** |
| Queue microbench, ordered add + remove (32 edges) | +3–4% (removals
+14–20%, additions −2–8%) |
| Concurrent end-to-end churn / `scope_stress` | within noise |
| Ordered end-to-end churn | noisy, no stable direction |
| `stress/fibonacci/500` | approximately +3–7% |

This is a tradeoff, not a blanket performance win. The single-threaded
microbench does not measure the old multi-threaded spin-wait. Part of
the Fibonacci cost comes from the extra upper lock required by the race
argument; reducing that cost without weakening correctness is a separate
follow-up. No new performance claim is made for the review follow-up.

### Limitations / status

- The verifier can only check placement using restored resident endpoint
metadata; absent/evicted endpoint numbers remain unknown, except root
placement can be checked without them.
- **Unmatched debt is silent.** Debt is only valid if the matching
addition is eventually generated and processed; producers must push
additions into a queue their operation runs to completion. This is a
requirement on producers, separate from the GC/snapshot exclusion rules
(debt does not pin tasks). Production aggregation drains are synchronous
(no async cancellation point) and now abort on panic rather than
returning a partial graph to a per-task catcher. This does not protect
jobs abandoned by a producer before the drain starts. If an addition
were lost, a later addition of the same task pair would pay the stale
debt instead of creating the edge, which can skew propagation. The old
retry loop panicked in that case; now only the
`verify_aggregation_graph` stop-time check reports residual debt (on
restored resident tasks).
- The stress test checks returned per-round counts. A stricter
exact-collectibles check also fails on unmodified canary under CPU load
and is tracked separately.
- A second separate-model correctness review found no new blocking
defect; its documentation note on the unmatched-debt assumption is
applied above. Neither review is a formal proof or approval.

### Removal benchmark and traversal follow-up

Two separate commits add permanent coverage and then the narrow
traversal change:

- `private::aggregation_queue` is an explicitly unstable native fixture
shared by benches and normal-library profiling. The opt-in
`TURBOPACK_BENCH_AGGREGATION_QUEUE=1` Criterion group runs ten routine
cases; `extended` exposes 756 filterable configurations:
inner/follower/mixed placement, true single-edge/one-ID batch/full-batch
jobs, references 1→0 or 2→1, data/no-data,
aligned/reversed/seeded-permuted IDs, and total sizes
1/8/16/32/33/64/66. Setup/restoration/checks are untimed. Raw edge
counts and aggregate contributions are checked after every phase.
Aggregation-number optimizations are deliberately disabled: this is
isolated queue work, not an end-to-end workload.
- `remove_follower_edges` consumes its input with existing
`TaskIdList::retain` swap-removal semantics. For aligned small linear
maps, visiting the swapped replacement avoids repeated scans. Both
result lists, edge multiplicity, positive-crossing effects, lock order,
debt prechecks and postchecks remain intact. Emitted batch order
intentionally changes; new real-handler tests cover Option/32/33 inputs,
ancestor propagation, retained references, duplicate IDs and mixed
positive/debt entries.

Normal-dependency profiling (not a backend test build) uses the same
frozen fixture, system allocator, one runtime/backend worker, no backing
storage, and no dev-only inline statistics. Release settings are
explicit (opt3/thin LTO/cgu1 plus the applicable regex-syntax size
override). Callgrind/DHAT cover 18 configurations; Criterion uses 30
samples, 1 s warmup, 3 s measurement and three interleaved parent/patch
rounds. Payload/context/queue construction and removal drain are timed;
restoration/assertions are not. Guest instructions and simulated events
are not hardware cycles.

| Synthetic removal case | Guest instructions | Median elapsed time |
|---|---:|---:|
| Aligned follower full batch, 32 edges | −9.79% | 1922.013 → 1743.011
ns (−9.31%) |
| Follower full batch, 64 edges | +5.23% | 3440.070 → 3670.363 ns
(+6.69%, first three rounds) |

The 64-edge case had non-overlapping slower 95% intervals in all three
rounds. Its extra confirmation pair was 3454.881 → 3569.930 ns (+3.33%),
still with non-overlapping intervals: the ≥5% magnitude did not repeat,
but a smaller slowdown persists. This tradeoff is retained and
disclosed, not described as regression-free. Full33 is +4.78%
elapsed/+5.99% instructions; mixed one-ID32 is +4.25% elapsed.
Mixed/permuted/retained cases can do more instruction work. Allocation
totals were equal across the three revisions in all 18 sampled cases.
Remaining normal-library helper/guard/accessor overhead relative to
canary is not fully diagnosed; symbol self costs do not prove extra
semantic locks. No blanket speedup claim.

Local verification before the canary rebase: all three full feature
modes pass 56/56 groups (204 unit tests plus one ignored benchmark);
10/10 normal + 10/10 graph-verified serial churn; fuzz seed 2019458061
completed 14,271 runs in 181 s without failure; both shared
addition-postcheck mutations fail the corresponding real-handler race
tests and were restored. Formatting, all-target clippy, private rustdoc
(warnings denied), fuzz manifest checks, 756-case extended smoke, ten
routine cases, disabled gating and the original ignored benchmark all
pass. Separate-model correctness and profiling-report reviews found no
remaining blocker; these are not human approval. Post-push CI is green
on `50e19c34`, including native cargo units/benches, miri, WASI/wasm,
Rust checks/rustdoc, and Windows tests; no reruns were needed.

### Canary rebase (`cc9ae702`)

Rebased the seven existing commits onto canary `cc65c414`. The only
manual integration resolves `TurboTasksBackend::stop`: keep canary's
`ActivityMonitor` shutdown handling, then run both graph and
aggregation-edge verification before dropping storage. All debt, race,
GC, helper and traversal behavior is unchanged; the benchmark and
traversal commits remain separate. Range-diff shows only this stop-hook
adaptation.

Both rebased layers were checked against exact verified source trees. On
this parent, full default, `verify_aggregation_graph` and
`aggregation_update_no_batch` suites each pass **59 test groups**,
including **216 unit tests** (one ignored benchmark). Formatting,
all-target clippy, private rustdoc with warnings denied and fuzz
manifest checks pass. The historical stress/fuzz/mutation and benchmark
results above were **not rerun on the new canary base**, and their
performance percentages are not new-base measurements. Post-push CI is
green on `cc9ae702`, including cargo units/benches, miri, WASI/wasm,
Rust checks/rustdoc and Windows tests; no reruns were needed.

### GC/no-resurrection canary rebase (`c2d937c1`)

Rebased the same seven commits onto canary `4a0609e1` (#100370). The
only manual resolution removes the obsolete `resurrect_deleted` import
while keeping aggregation-edge helpers. Upstream's live-task assertions,
deleted-candidate filtering, GC task-cache removal and optimization
changes remain intact. The previous `ActivityMonitor` stop-verification
integration is retained. No debt, race, balancing or traversal logic was
weakened; final source exactly matches the verified integration tree.

Latest local verification: full default, `verify_aggregation_graph` and
`aggregation_update_no_batch` suites pass **59 groups each**, including
**220 unit tests** (one ignored benchmark), the updated GC
reconnect/replacement tests and all four `task_by_type` tests.
Formatting, all-target clippy, private rustdoc with warnings denied and
fuzz manifest check pass. On the child an existing 16-bit random-tag
re-execution oracle collided once despite correct 1→2 propagation and
logged re-execution; targeted and final full runs passed, with a
deterministic-oracle follow-up recorded. No test assertions were
removed. Older performance, fuzz and stress numbers above are
historical, not measurements on this base. Post-push CI is green on
`c2d937c1`, including cargo units/benches, miri, WASI/wasm, Rust
checks/rustdoc and Windows tests; no CI reruns were needed.

### Reviewer follow-up (`614290c4`, `75538d53`)

- Added a narrow fatal-panic boundary to production aggregation queue
drains, including stats and GC entry points. The manager's outer abort
wrapper does not by itself protect backend calls reached inside the
per-task panic catcher. Native subprocess tests first reproduced
containment of an injected panic after a debt write, then verified
process abort and normal controls. User-code panic policy is unchanged;
synchronous drains have no async cancellation point, and pre-drain job
abandonment remains a producer responsibility.
- Shortened GC docs; renamed follower-edge propagation and activeness
state; used `filter_map`; clarified the outgoing-diff merge TODO without
implementing merging.
- Extracted optimization heuristics without changing raw debt-inclusive
counts, single power-of-two versus batch ceiling-power triggers, caps or
pending flags. Added boundary/debt-count regressions.
- Kept singleton peeling and Option/Vec job shapes. The shared iterator
helper has no-storage and ready-task paths that do not allocate restore
batches; no blanket allocation or speedup claim is made. Existing
real-handler tests cover root/root placement and root debt payment.

Local checks on both layers: three full feature modes pass **59 groups
each** (parent **223** units, child **229**, one ignored benchmark);
stats libraries, fmt, all-target clippy, private rustdoc with warnings
denied, and fuzz manifest checks pass. Both layers pass opt-in benchmark
smoke: **10 routine + 756 extended cases**, with default omission
verified. These are correctness smoke tests, not fresh performance
measurements. The historical timings/stress/fuzz results above were not
rerun for this update. Pushed-tip CI is green on `75538d53`, including
cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows
tests; this parent's CI needed no reruns. Review threads remain
unresolved for the reviewer.

<!-- NEXT_JS_LLM -->

<!-- fleet e31fbff5-6610-4d8e-9172-b2528e8725ad -->

---------

Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
sokra added a commit that referenced this pull request Nov 10, 2026
…s and balancing (#100892)

Stacked on #100873 (negative follower debt). Only the commits on top of
#100873 belong to this PR.

### What?

Within one `AggregationUpdateQueue`, aggregation edge removals
(`InnerOf{Upper,Uppers}LostFollower(s)`) now wait in a separate removal
queue until all ordinary jobs, aggregation number updates and balancing
in that queue are done.

### Why?

Previously additions and removals were interleaved in one FIFO. When an
edge (or a task reachable through another edge) was removed and re-added
in the same queue, its aggregated data went `1 → 0 → 1`: data was
removed and re-added, collectibles dependents were invalidated, dirty
containers flipped and readers could briefly miss collectibles. With
removals last it goes `1 → 2 → 1`, so none of that temporary work
happens.

### How?

- `push` routes the three lost-follower jobs into `removal_jobs`;
everything else (data updates, active/parent counts, invalidations,
additions) keeps its queue and order.
- `process` priority: `jobs` → aggregation number updates → balancing →
**removals** → optimize → find dirty → schedule. A small `pop_job`
helper picks a removal only when the higher-priority queues are empty,
so work generated by a removal runs before the next removal, and
generated removals append FIFO. The existing job dispatch is unchanged.
- GC: `execute_until_rebalance` must drain removals while balancing
stays deferred. It moves balance jobs out of the queue after every step
and returns both previously pending and newly generated ones; the
caller's deleted-endpoint filtering is unchanged.
- Safety: a removal processed later is the ordinary case. A removal that
still overtakes its addition (another queue, or an addition generated
later) records follower debt as in #100873. The PR relies on the
existing debt contract and doesn't weaken it. No new locking, no schema
change.

### Scope / limitations

- Queue-local only: helps when the keeping addition/balance is already
pending in the same queue. No atomicity across separate queues or
concurrent operations.
- Task completion still removes old children (`cleanup_old_edges`) and
connects new children (`connect_children`) in separate queues, so a task
switching between children that share a descendant still churns.
Combining those is planned as a separate, later stacked PR after
measuring this one.

### Tests

New `deferred_removal_tests` (real backend, step-by-step queue
processing):
- same-edge replacement and a moved shared descendant: the upper's
collectible never drops to zero at any step, and a collectibles reader
is not invalidated;
- genuine loss still removes data and invalidates the reader;
- removals only run when no ordinary job, aggregation number update or
balance is pending;
- work generated by a removal runs before the next removal, generated
removals append FIFO;
- `execute_until_rebalance` drains removals but executes no balancing
and returns pending and newly generated balance edges.

An existing debt test that mixed remove/remove/add in one queue now uses
separate queues so it still forces a real overtake. Mutations: routing
removals back into the ordinary FIFO fails 4 of the new tests; not
collecting newly generated balances in the GC path fails the GC test.

### Verification

- `cargo test -p turbo-tasks-backend` in default,
`verify_aggregation_graph`, and `aggregation_update_no_batch`: 56/56
test groups each (including the existing stress tests).
- `aggregation_churn` stress: 20/20 serial runs normal and 20/20 with
graph verification.
- `fuzz_graph`: 8,543 runs in 181 s, no crashes.
- clippy (`--all-targets`, default and
`verify_aggregation_graph,trace_aggregation_update_stats`), fuzz crate
clippy, fmt, private rustdoc with `-D warnings`.

### Work counts and performance

Exact counts (`replace_work`: 32 existing edges removed and re-added in
one queue, a reader on the upper; feature
`trace_aggregation_update_stats`), vs #100873:

| | aggregated data update jobs | collectibles invalidation jobs |
|---|---:|---:|
| #100873 | 2 | 2 |
| this PR | 0 | 0 |

Queue microbench (release unit-test binary, 3 interleaved same-machine
rounds, Criterion central estimates):

| case | #100873 | this PR |
|---|---|---|
| replace (remove + re-add in one queue) | 12.7–12.9 µs | 8.5–10.2 µs
(−20…−35%) |
| overtake (removals in a separate earlier queue) | 10.2–10.4 µs |
11.0–12.3 µs (+6…+21%, noisy) |
| ordered / ordered additions / ordered removals | — | within noise,
removals possibly slightly slower |

The microbench binary includes inert test-only hooks on this branch, so
it does not isolate production overhead. End-to-end benches
(`aggregation_churn`, `stress/fibonacci/500`) drifted too much across
rounds to draw a conclusion either way. This is not claimed as a general
speedup.

### Rebase onto removal benchmark/traversal follow-up

Rebased the existing two logical commits onto #100873 at `50e19c34`,
retaining the deferred-removal change and separate balance-queue drain
refinement. The parent's permanent removal fixture/benchmarks and
swap-order traversal are inherited, not additional child scope. Only the
new parent's test observer needed adapting to inspect `removal_jobs`
rather than the ordinary queue; production priority, GC balance
collection and drain behavior are unchanged by the rebase.

Local verification before the canary rebase: full default,
`verify_aggregation_graph` and `aggregation_update_no_batch` suites each
pass **56/56 groups** (210 unit tests plus one ignored benchmark). This
includes the parent's 756-configuration fixture test and
traversal/multiplicity/ancestor regressions, the actual-handler race
tests, and this child's priority, genuine-loss, no-invalidation and
GC-drain tests. Formatting, all-target clippy, private rustdoc with
warnings denied and fuzz manifest check pass. The older
stress/fuzz/performance numbers above are historical, not newly measured
on this rebased child. Parent normal-library profiling explicitly
reports a 32-edge benefit and a larger-map tradeoff; it is not evidence
of an end-to-end child speedup. Post-push CI is green on `c26cd62d`,
including native cargo units/benches, miri, WASI/wasm, Rust
checks/rustdoc, and Windows tests; no reruns were needed.

### Canary restack (`e9607690`)

Replayed the existing two child commits onto #100873 at `cc9ae702`
(canary base `cc65c414`). Both child patches are unchanged by
range-diff. Queue priority, GC balance collection and separate drain
refinement are preserved. The final source tree exactly matches the
locally verified integration tree.

On this child, full default, `verify_aggregation_graph` and
`aggregation_update_no_batch` suites each pass **59 test groups**,
including **222 unit tests** (one ignored benchmark). Formatting,
all-target clippy, private rustdoc with warnings denied and fuzz
manifest checks pass. Prior stress/fuzz/performance results above remain
historical; no new-base performance or end-to-end speedup is claimed.
Post-push CI is green on `e9607690`, including cargo units/benches,
miri, WASI/wasm, Rust checks/rustdoc and Windows tests; no reruns were
needed. Base relationship and ready status are unchanged; higher layers
were not modified.

### GC/no-resurrection canary restack (`d91b5264`)

Restacked the same two child commits onto #100873 at `c2d937c1` (canary
base `4a0609e1`, #100370). Both child patches remain unchanged by
range-diff: removal priority, deferred GC balance collection and the
separate drain refinement are preserved. Upstream no-resurrection
behavior and deleted-endpoint filtering are retained. The final source
exactly matches the verified integration tree.

Latest local verification: full default, `verify_aggregation_graph` and
`aggregation_update_no_batch` suites pass **59 groups each**, including
**226 unit tests** (one ignored benchmark), updated GC
reconnect/replacement and `task_by_type` tests, plus this child's
GC/priority regressions. Formatting, all-target clippy, private rustdoc
with warnings denied and fuzz manifest check pass. Initial default
`recompute_dependency` hit an existing 16-bit random-tag collision: both
functions logged re-execution and value correctly changed 1→2, but tags
were equal. A targeted rerun and final full runs pass; a
deterministic-oracle follow-up is recorded, without changing test or
production behavior here. Older stress/fuzz/performance figures remain
historical; no new-base speedup is claimed. Post-push CI is green on
`d91b5264`, including cargo units/benches, miri, WASI/wasm, Rust
checks/rustdoc and Windows tests; no CI reruns were needed. Base, ready
status and higher stack layers are unchanged by this update.

### Restack onto reviewer follow-up (`fd4345e9`)

Replayed only the existing two logical child commits onto #100873 at
`75538d53`. The GC drain keeps its removal-job predicate, deferred
balance collection and per-step `drain(..)` inside the parent's new
fatal-panic boundary. Final source matches the verified child
integration exactly. Queue priority, debt pre/postchecks,
deleted-endpoint filtering, PR base/readiness and higher stack branches
are unchanged.

Local verification: default, `verify_aggregation_graph` and
`aggregation_update_no_batch` full suites each pass **59 groups**,
including **229 unit tests** (one ignored benchmark); stats library
passes (two ignored benchmarks), including ordinary/stats/GC fatal-panic
subprocess tests. Formatting, all-target clippy, private rustdoc with
warnings denied and fuzz manifest checks pass. Opt-in queue benchmark
smoke passes **10 routine + 756 extended cases**, and default omission
is checked. Older timing/stress/fuzz figures above remain historical; no
new speedup is claimed. Pushed-tip CI is green on `fd4345e9`, including
cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows
tests. The first cargo-unit run had one
`turbopack-go::live::versioned_dependencies_use_the_module_cache_without_downloads`
failure with `Permission denied (os error 13)`; one rerun passed without
source changes. That test's error-context/permission investigation is
tracked separately.

<!-- NEXT_JS_LLM -->


<!-- fleet e31fbff5-6610-4d8e-9172-b2528e8725ad -->

---------

Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Nov 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

created-by: Turbopack team PRs by the Turbopack team. locked Turbopack Related to Turbopack with Next.js.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ 换网址