Skip to content

Stop timed-out watcher setups from rescanning large repos every 5 minutes - #9828

Merged
paseo-bot[bot] merged 3 commits into
mainfrom
fix/9822-abort-timed-out-watcher-setup
Nov 16, 2026
Merged

paseo-bot[bot] merged 3 commits into
mainfrom
fix/9822-abort-timed-out-watcher-setup

Conversation

@paseo-bot

@paseo-bot paseo-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Linked issue

Closes #9822

Refs #9611 (a macOS workspace whose watcher fails on the 250,000-entry limit and rescans every ~5 minutes; that failure is not a setup timeout, so the timeout backoff here does not cover it)

Type of change

  • Bug fix

Reasoning

On a busy host, a large repo's working-tree or repository-metadata watcher can take longer than the 10 second setup deadline. The daemon gave up waiting at the deadline but let the startup scan keep walking the whole tree, attaching a watcher per directory, and then tore it down. Recovery retried on a ladder capped at 5 minutes, so every timed-out directory ran another full scan every 5 minutes for as long as the host stayed busy, which kept the daemon's main thread stalled and made switching chats slow.

Before and after, on a workspace whose watcher setup misses the deadline:

  • Before: the scan runs to completion after the deadline, and a fresh full scan starts every 5 minutes indefinitely.
  • After: the scan stops at the deadline. Consecutive timeouts double the retry delay up to one hour, and a watcher that sets up in time resets it. Degraded polling keeps the checkout fresh in the meantime, on its existing cadence.

This builds on @Spshulem's commit from #9818, which found the cause and made both changes, and adds one fix: an abort that landed while the observation was still reading the root closed it before the backend started, and the backend then started anyway. On macOS and Windows that left the root watcher open with nothing to close it.

Goals

  • A watcher setup that misses its deadline stops scanning.
  • Repeated setup timeouts on the same directory back off up to one hour; an accepted subscription resets the backoff.
  • Aborting a subscription at any point during setup leaves no watcher open.

Non-goals

QA

  • Real Linux file observer on a 4,200-directory tree, with the deadline simulated 5 ms into setup. On main the scan kept running about 120 to 130 ms past the deadline and the subscription resolved anyway (two runs). With this branch it rejected 0.1 ms after the deadline (two runs). On the reported busy host the same scan runs past 10 seconds.
  • The tests from fix(server): abort timed-out watcher setups and back off retries per directory #9818 fail on main's sources (8 failures) and pass with the fix: aborting a pending subscription stops its startup scan, timed-out working-tree and repository-metadata setups are aborted and back off to an hour while degraded polling continues, and an accepted subscription resets the backoff.
  • The abort test in file-observer/index.test.ts now aborts 10 ms into setup on a tree past the Linux backend's 5,000-directory limit. With the abort listener removed it fails on the directory-limit error (2/2 runs); with it, it passes (3/3).
  • New file-observer/internal/observation.test.ts: closing an observation before its native backend starts leaves no watcher open. It fails without the guard in Observation.start() (one watcher left open) and passes with it.
  • file-observer and workspace-git-service suites: 228/228. Other server suites that use the file observer or WorkspaceGit (checkout diff, diff subscriptions, worktree session, workspace runtime store, auto-archive, MCP server, project suggestions, session workspaces): 384 passed, 4 skipped.
  • Not run: macOS FSEvents and Windows backends on real hardware, and a CPU-starved host. The native backend leak is covered with a fake watcher on Linux.

Risk

  • A directory whose setup keeps timing out now recovers its watcher up to an hour later than before, instead of within 5 minutes. Changes still show up through degraded polling during that time.
  • FileObserver.subscribe gains an optional signal; callers that don't pass one behave as before.

Checklist

  • One focused change
  • npm run typecheck passes
  • npm run lint passes
  • npm run format passes
  • QA evidence
  • Tests added or updated where it made sense

Supersedes

🤖 Generated with Claude Code

Spshulem and others added 2 commits November 16, 2026 05:15
A watcher subscription that missed WORKSPACE_GIT_WATCHER_SUBSCRIBE_TIMEOUT_MS
was abandoned but its startup scan kept walking the tree to completion, and
recovery retried every five minutes forever, repeating the full scan each
time. On a CPU-starved host that kept dozens of directories rescanning.

- FileObserver.subscribe accepts an abort signal that stops the startup scan
  and rejects; WorkspaceGit aborts setups at the deadline and on dispose.
- Consecutive setup timeouts extend the recovery delay past the 5-minute
  ladder cap, doubling up to one hour; an accepted subscription resets it.
  Degraded polling keeps its 5-60 s cadence meanwhile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aborting a subscription closes its observation, which can happen while
start() is still reading the root. The backend then started after its close
had finished, and the native backend opened a root watcher that nothing
would close.

Co-authored-by: Spencer Shulem <1703265+Spshulem@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@paseo-bot paseo-bot Bot added the diagnosing The diagnose bot is working on this issue label Nov 16, 2026
@paseo-bot

paseo-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Confidence 5/5

The fix works and both review passes found no open issues at this commit. The one finding from the first review, that the abort test passed with the scan-stopping abort removed, is fixed in 3c48abb. The test now builds more directories than the Linux backend watches, so a scan that keeps walking after the abort fails on the directory limit instead of rejecting with the abort reason.

Summary

Before: on a busy host, a large repo whose watcher setup missed the 10 second deadline kept running its full startup scan after the deadline, and retried a full scan every 5 minutes for as long as the host stayed busy.

After: the scan stops at the deadline, and retries double from 30 seconds up to one hour while timeouts continue. Degraded polling keeps the checkout fresh in the meantime. A setup that completes in time returns the directory to the normal 5 minute retry ceiling.

Product changes

  • A workspace whose watcher setup keeps timing out retries setup at 30s, 1m, 2m, 4m, 8m, 16m, 32m, then hourly, instead of capping at 5 minutes.
  • A timed-out watcher setup stops its directory scan at the deadline instead of finishing it in the background.
  • FileObserver.subscribe (server-internal) accepts an optional signal that aborts setup and rejects. Callers that do not pass one behave as before.

Size

Proportional. The abort reuses the existing backend checks that stop work once an observation is closed, so the observer change is one abort listener plus one guard in Observation.start(). The backoff adds one counter to the existing recovery state and does not add a second recovery mechanism.

Category Added Removed
Production 57 6
Tests and fixtures 193 0
Generated artifacts (including lockfiles) 0 0
Docs 1 1
Total 251 7

What could break

No open findings.

What I verified

Run in this review, on Linux, at 3c48abb:

  • file-observer/index.test.ts and file-observer/internal/observation.test.ts: 16/16 pass. The new abort test passed 8 of 8 reruns.
  • workspace-git-service.test.ts, filtered to the new backoff tests: 4/4 pass.
  • Scratch tests against the real Linux observer covered these cases, and none left an observation, native handle, or callback behind:
    • the signal already aborted at the call
    • an abort while the root is being read
    • an abort during the scan
    • two subscriptions on the same root with one aborted (the other stays active and keeps receiving events)
  • Scratch test, native-recursive backend (macOS/Windows path) with a fake watcher, closed mid-audit: no watcher left open.
  • Fix check for the previous thread: the test tree has 5,681 directories, above the Linux backend limit of 5,000. The unaborted scan takes about 107 ms to hit the limit locally, against the 10 ms abort in the test.

Source-traced:

  • Nothing can interleave between start() resolving and subscribe() returning, because the deadline and dispose aborts fire from timer callbacks.
  • In WorkspaceGit the late rejection is handled, the settle callback still runs, and the dispose listener and timeout are cleared per attempt.
  • Backoff state is kept per working-tree directory and per repository .git root, advances once per failed attempt, resets only on an accepted subscription, and its timer is cleared on target close and service dispose.
  • Degraded polling reschedules while the target has no subscription, independent of the recovery timer. The backoff test asserts at least 59 reads during the capped hour.
  • WorkspaceGit is the only production caller that passes a signal.

Not blocking, not posted as threads:

  • Closing a watch target (archiving the workspace or removing its last listener) while its setup is still pending does not abort the scan. The scan runs until it completes or the 10 second deadline aborts it. This is unchanged from main and bounded by the deadline.
  • After a run of timeouts, a different setup failure (for example EMFILE) keeps the longer timeout delay, because the timeout count resets only on an accepted subscription. The code comment states this.
  • subscribe() rejects with a filesystem error instead of the abort reason when the abort coincides with that error (reproduced with a missing root). WorkspaceGit has already rejected with its own timeout error and ignores the late result.

Not verified: macOS FSEvents and Windows backends on real hardware, and a CPU-starved host.

Reviews (2) · Last reviewed commit: 3c48abb · Reviewed by Paseo

The test aborted before the backend started, so it passed with the abort
listener removed. A tree past the Linux directory limit makes a scan that
keeps walking fail on the limit instead of rejecting with the reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@paseo-bot paseo-bot Bot added diagnosed The diagnose bot investigated this; on a PR, the bot authored it and removed diagnosing The diagnose bot is working on this issue labels Nov 16, 2026
@paseo-bot
paseo-bot Bot merged commit 3325974 into main Nov 16, 2026
21 checks passed
@paseo-bot
paseo-bot Bot deleted the fix/9822-abort-timed-out-watcher-setup branch November 16, 2026 09:10

@paseo-bot paseo-bot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merged as 3325974 (squash). The closing keyword closed #9822, and I closed the superseded #9818 with credit to its author, whose commit this branch carries.

Product delta: on a busy host with a large repo, a watcher setup that misses its 10 s deadline now stops scanning at the deadline. Before, the scan ran to completion anyway and started again every 5 minutes for as long as the host stayed busy. Consecutive setup timeouts now back off 30, 60, 120, 240, 480 s and on up to 1 h, and degraded polling keeps the Changes view fresh in the meantime. A watcher that sets up in time resets the backoff. Other watcher failures keep the existing 30 s to 5 min ladder.
Protocol delta: none. FileObserverOptions.signal is a server-internal, optional parameter.

What it does

A timed-out or disposed watcher setup in WorkspaceGit now aborts the file observer's startup scan, through a signal passed to FileObserver.subscribe. Consecutive setup timeouts get their own backoff counter in the existing recovery state. Observation.start() also no longer starts the native backend after an abort has already closed the observation.

Review

  1. Does it fix what was reported? Yes. On main, subscribeWithDeadline raced the 10 s timer but could not stop FileObserver.subscribe. The Linux scan loop only stops when the observation closes, and nothing closed it during start(). Retries plateaued at 300 s because a timed-out setup never set establishedAt. On a CPU-starved isolated daemon, a real 4,500-directory repo showed this: on the merge base, inotify watches climbed 339 → 2028 → 3750 after the timeout; on the head, the count never went above 16.
  2. Is it the right fix? Yes. Cancellation is now a capability of the observer, the deadline stays with WorkspaceGit, and the backoff is one counter in the existing WatchRecoveryState. The Observation.start() guard sits in the layer that owns the observation's lifecycle. Nothing compensates for the scan from the outside.
  3. Does it do anything else? No. There is no protocol, schema, setting, or dependency change. The longer recovery ceiling, up to 1 h instead of 5 min, is the behavior the issue asked for. The diagnose bot raised it as a question that can be answered after merging. Changes keep appearing through degraded polling during that time.
  4. Is it worth shipping? Yes. It has about 57 production lines across two server files plus one guard. It affects anyone with a large repo on a loaded host.

The only other caller of the file observer, diff/dependencies.ts, passes no signal, so its behavior is unchanged.

QA checklist

Every item ran on an isolated daemon built from this branch, against a real ~4,500-directory, ~67,500-file git repo, with busy loops pinned to the daemon's CPU. Items were run head against merge base where noted.

  1. Reported scenario, base vs head: pass. Base: after the timeout, watches rose to 3,750 before teardown. Head: the count never went above 16, and nothing rose after the timeout.
  2. Retry cadence under sustained load: pass. The six timeout lines were 46 s, 77 s, 137 s, 256 s, and 496 s apart. That is the 30/60/120/240/480 s ladder plus the 10 s deadline and about 6 s of lag. The base caps at 300 s.
  3. Degraded polling keeps Changes fresh while setups time out: pass. A file change appeared with no watcher running.
  4. Recovery after the load lifts: pass. The watcher recovered with 4,537 watches, and an edit appeared in under 1 s.
  5. Callers that reach the user: pass. With no load, create, edit, and delete each showed in the Changes view in about 340 ms. A file made by a real Codex session also appeared in Changes.
  6. Shutdown or archive while a setup is pending: pass.
    • SIGTERM about 5 s into a pending setup, on head: graceful exit in 4.3 s, no forced kill, no leftover processes or watches.
    • Same sequence on the merge base, two runs: the worker was force-killed both times, after 17 s and 11 s.
    • Archive about 5 s into a pending setup, on head: no retry afterwards, no leftover watches, no polling.
  7. Regression tests red without the fix, green with it: pass. The head passes 134/134. With the base sources, 9 fail. The file-observer suite passes 32/32 and the server typecheck is clean.
  8. Native-recursive backend with real fs.watch, close during the stat phase: pass. The base left 72,071 inotify watches and the process hung. The head left 0.

Outside the checklist

  • Archive during a pending setup still logs a timeout line. It printed about 5 s after the archive. Archive does not cancel a setup that is already in flight, on main or here. With this branch, that setup is now aborted at its deadline and never retried, so the cost is bounded at 10 s. This is unchanged from main and has no user-visible effect, so I considered it and did not count it.
  • project create timed out once. It hit a 30 s git status timeout on a freshly copied repo under deliberate CPU starvation, and the retry succeeded. The PR does not touch that path, and the failure came from the test load. Considered, not counted.
  • The Files tab tree did not update live. That listing has no live subscription, and the PR does not touch it. Considered, not counted.

Review threads

The one thread (an abort test that passed without the listener) was settled by 3c48abb and is resolved. Item 7 confirmed the red/green.

Reviewed commit: 3c48abb

@paseo-bot paseo-bot Bot added the community-notified Merged work has been communicated to the people who reported or discussed it label Nov 16, 2026
x9x9x9x9x9x91 pushed a commit to x9x9x9x9x9x91/paseo that referenced this pull request Nov 18, 2026
…utes (getpaseo#9828)

* fix(server): abort timed-out watcher setups and back off their retries

A watcher subscription that missed WORKSPACE_GIT_WATCHER_SUBSCRIBE_TIMEOUT_MS
was abandoned but its startup scan kept walking the tree to completion, and
recovery retried every five minutes forever, repeating the full scan each
time. On a CPU-starved host that kept dozens of directories rescanning.

- FileObserver.subscribe accepts an abort signal that stops the startup scan
  and rejects; WorkspaceGit aborts setups at the deadline and on dispose.
- Consecutive setup timeouts extend the recovery delay past the 5-minute
  ladder cap, doubling up to one hour; an accepted subscription resets it.
  Degraded polling keeps its 5-60 s cadence meanwhile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(file-observer): keep an aborted observation's backend from starting

Aborting a subscription closes its observation, which can happen while
start() is still reading the root. The backend then started after its close
had finished, and the native backend opened a root watcher that nothing
would close.

Co-authored-by: Spencer Shulem <1703265+Spshulem@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(file-observer): abort a subscription mid-scan in the abort test

The test aborted before the backend started, so it passed with the abort
listener removed. A tree past the Linux directory limit makes a scan that
keeps walking fail on the limit instead of rejecting with the reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Spencer Shulem <spshulem@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paseo-bot[bot] <266920839+paseo-bot[bot]@users.noreply.github.com>
Co-authored-by: Spencer Shulem <1703265+Spshulem@users.noreply.github.com>
(cherry picked from commit 3325974)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-notified Merged work has been communicated to the people who reported or discussed it diagnosed The diagnose bot investigated this; on a PR, the bot authored it

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Watcher setup that misses its deadline keeps scanning large repos and repeats every ~5 minutes, stalling the daemon on busy Linux hosts

1 participant

⌂ 换网址