Skip to content

coordinator winner retirement: reader handover, stream draining and two-ended arbitration #313

Open

Description

@dirvine

Follow-up to #310. The #310 PR moves the hole-punch duplicate check under
the lifecycle write lock: two candidate winners can no longer both pass
the check and orphan each other. That PR deliberately leaves the
accepted-coordination insert as it is on master. It replaces the
winner-map slot under the lifecycle lock and closes nothing. This issue
tracks retiring the displaced connection safely.

1. The coordinator insert orphans a displaced untracked winner (#310,
bullet 2).
establish_connection_to_validated_candidate_inner inserts
the dialled connection C into connections without lifecycle
registration. A hole-punch winner H that it displaces is not in the
lifecycle map either. H's detached close and observed-address watchers
keep it open until the peer closes it, the idle timeout fires or the
endpoint shuts down.

2. Closing H at the insert is wrong (Codex review of #310, P2-1). H can
be the connection the application reads from:

  • await_hole_punch_outcome obtains H, starts its reader and returns
    success (src/p2p_endpoint.rs ~5600/5613/5630).
  • When the in-flight coordination dial then completes as C, C gets only
    observed-address and close watchers. It is not queued for
    accept_connection, and the event callback reports only progress
    (p2p_endpoint.rs ~2944/3120).
  • H's reader exit takes ReaderExitOutcome::Noop (nat_traversal_api.rs
    handle_reader_exit; p2p_endpoint.rs ~10496). The outer peer state
    stays connected.

So closing H kills live traffic and unfinished streams, and no reader
takes over. "Untracked" does not mean "unused". The #310 PR has a
regression guard for this:
coordinator_dial_leaves_a_displaced_untracked_winner_open_and_usable.

3. A locked custody check is not enough (Codex P2-2). Interleaving:

  1. A caller keeps H from get_connection.
  2. The coordinator replaces H with C, sees that H is untracked, and
    releases the lock.
  3. Before the close, another thread calls add_connection(peer, H). This
    records H as Live and puts it back in the winner map.
  4. The stale decision then closes a tracked winner. C can also become an
    untracked orphan through the registration insert.

Checking custody again outside the lock still races. Retirement needs an
atomic claim that registration respects, or a lifecycle transition that
stops resurrection between the decision and the close.

4. The two ends can disagree (Codex answer 2). Each end's choice of
untracked winner is local:

  • hole-punch: the first winner keeps the slot;
  • coordinator: the last insert takes the slot.

Neither is coordinated with the remote end's lifecycle tiebreak
(candidate_wins, register_connection_lifecycle_parts). For example,
local hole-punch first-wins can reject a second connection that the
remote's same-initiator registration chooses as its newer generation.
This predates #310. Rejecting every coordinator newcomer does not fix it
either.

5. The same orphan class exists at two more sites. The
lifecycle-registration insert (register_connection_lifecycle_parts, ~7473
on the #310 base) and the repromotion insert
(repromote_surviving_connection, ~7984 on the #310 base) can displace a
different untracked value without closing it. Any fix must keep two cases
distinct:

  • tracked Superseded generations, kept open for stream draining;
  • reinsertion of the same handle (same stable_id).

Possible direction (not decided):

  • Route hole-punch and accepted-coordination winners through lifecycle
    registration. They then get generations, the same-initiator tiebreak and
    the Superseded drain window. Alternatively, add an atomic retirement claim
    that registration respects.
  • Hand the reader over to the replacement before the old winner is retired,
    for example through pending_accepts or the connection_promoted_tx
    promotion signal.
  • Keep a drain window for the old winner's streams.

Acceptance tests:

  • Finalize H at the P2P layer, with a P2pEndpoint reader running, before
    releasing the coordinator dial. Check ongoing and later traffic and the
    outer connected_peers state (P2-1).
  • Add a gate between the retirement decision and the close, and run a
    concurrent add_connection(H) there. Neither a tracked winner closed nor
    an orphaned C is allowed (P2-2).
  • Same-peer, two-ended regression: both ends run the accepted coordination
    and the hole-punch race, and both end on the same live connection
    (answer 2).
  • An untracked value displaced at the registration or repromotion insert
    is either closed or tracked, never orphaned. Superseded drain survivors
    and same-handle reinsertion are left alone.

References:

Activity

  1. dirvine commented on Oct 9, 2026

    @dirvine
    CollaboratorAuthor

    #315 now depends on this issue. The parked fix makes an inbound winner actively close the superseded generation. Codex found that in the coordinator race this lets the remote end close the connection that carries A's only application reader, because A's coordinator path starts no reader for C. So the reader handover here is a prerequisite for #315.

  2. added a commit that references this issue on Oct 9, 2026
  3. dirvine commented on Oct 9, 2026

    @dirvine
    CollaboratorAuthor

    Part 1 merged in #318 (master da719d0). Raw coordinator and hole-punch winners, plus materialize, now get a reader. That fixes the B→A non-delivery Codex confirmed in the coordinator race. Each connection also gets exactly one reader owner, kept until the reader has stopped. It adds no new close and makes no routing, lifecycle, wire or public-API change. Part 2 is still open and parked as an ADR-level decision: retiring displaced or superseded connections, and arbitration between the two ends. Plan and reviews: x0x .planning/team-2026-10-05/s-313-plan.md, and the s-313-plan-review-codex.md and s-aq313-p1-review-codex.md reports. Release: 0.27.57 is being prepared (D234).

  4. added 2 commits that reference this issue on Oct 9, 2026
  5. dirvine commented on Oct 9, 2026

    @dirvine
    CollaboratorAuthor

    Part 1 (#318) is released in ant-quic 0.27.57 (crates.io, 2026-10-09). Part 2, retirement and two-ended arbitration, is still parked.

  6. added 6 commits that reference this issue on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址