Repository navigation
Stop timed-out watcher setups from rescanning large repos every 5 minutes - #9828
Conversation
A watcher subscription that missed WORKSPACE_GIT_WATCHER_SUBSCRIBE_TIMEOUT_MS was abandoned but its startup scan kept walking the tree to completion, and recovery retried every five minutes forever, repeating the full scan each time. On a CPU-starved host that kept dozens of directories rescanning. - FileObserver.subscribe accepts an abort signal that stops the startup scan and rejects; WorkspaceGit aborts setups at the deadline and on dispose. - Consecutive setup timeouts extend the recovery delay past the 5-minute ladder cap, doubling up to one hour; an accepted subscription resets it. Degraded polling keeps its 5-60 s cadence meanwhile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aborting a subscription closes its observation, which can happen while start() is still reading the root. The backend then started after its close had finished, and the native backend opened a root watcher that nothing would close. Co-authored-by: Spencer Shulem <1703265+Spshulem@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confidence 5/5The fix works and both review passes found no open issues at this commit. The one finding from the first review, that the abort test passed with the scan-stopping abort removed, is fixed in 3c48abb. The test now builds more directories than the Linux backend watches, so a scan that keeps walking after the abort fails on the directory limit instead of rejecting with the abort reason. SummaryBefore: on a busy host, a large repo whose watcher setup missed the 10 second deadline kept running its full startup scan after the deadline, and retried a full scan every 5 minutes for as long as the host stayed busy. After: the scan stops at the deadline, and retries double from 30 seconds up to one hour while timeouts continue. Degraded polling keeps the checkout fresh in the meantime. A setup that completes in time returns the directory to the normal 5 minute retry ceiling. Product changes
SizeProportional. The abort reuses the existing backend checks that stop work once an observation is closed, so the observer change is one abort listener plus one guard in
What could breakNo open findings. What I verifiedRun in this review, on Linux, at 3c48abb:
Source-traced:
Not blocking, not posted as threads:
Not verified: macOS FSEvents and Windows backends on real hardware, and a CPU-starved host. Reviews (2) · Last reviewed commit: 3c48abb · Reviewed by Paseo |
The test aborted before the backend started, so it passed with the abort listener removed. A tree past the Linux directory limit makes a scan that keeps walking fail on the limit instead of rejecting with the reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Merged as 3325974 (squash). The closing keyword closed #9822, and I closed the superseded #9818 with credit to its author, whose commit this branch carries.
Product delta: on a busy host with a large repo, a watcher setup that misses its 10 s deadline now stops scanning at the deadline. Before, the scan ran to completion anyway and started again every 5 minutes for as long as the host stayed busy. Consecutive setup timeouts now back off 30, 60, 120, 240, 480 s and on up to 1 h, and degraded polling keeps the Changes view fresh in the meantime. A watcher that sets up in time resets the backoff. Other watcher failures keep the existing 30 s to 5 min ladder.
Protocol delta: none. FileObserverOptions.signal is a server-internal, optional parameter.
What it does
A timed-out or disposed watcher setup in WorkspaceGit now aborts the file observer's startup scan, through a signal passed to FileObserver.subscribe. Consecutive setup timeouts get their own backoff counter in the existing recovery state. Observation.start() also no longer starts the native backend after an abort has already closed the observation.
Review
- Does it fix what was reported? Yes. On main,
subscribeWithDeadlineraced the 10 s timer but could not stopFileObserver.subscribe. The Linux scan loop only stops when the observation closes, and nothing closed it duringstart(). Retries plateaued at 300 s because a timed-out setup never setestablishedAt. On a CPU-starved isolated daemon, a real 4,500-directory repo showed this: on the merge base, inotify watches climbed 339 → 2028 → 3750 after the timeout; on the head, the count never went above 16. - Is it the right fix? Yes. Cancellation is now a capability of the observer, the deadline stays with
WorkspaceGit, and the backoff is one counter in the existingWatchRecoveryState. TheObservation.start()guard sits in the layer that owns the observation's lifecycle. Nothing compensates for the scan from the outside. - Does it do anything else? No. There is no protocol, schema, setting, or dependency change. The longer recovery ceiling, up to 1 h instead of 5 min, is the behavior the issue asked for. The diagnose bot raised it as a question that can be answered after merging. Changes keep appearing through degraded polling during that time.
- Is it worth shipping? Yes. It has about 57 production lines across two server files plus one guard. It affects anyone with a large repo on a loaded host.
The only other caller of the file observer, diff/dependencies.ts, passes no signal, so its behavior is unchanged.
QA checklist
Every item ran on an isolated daemon built from this branch, against a real ~4,500-directory, ~67,500-file git repo, with busy loops pinned to the daemon's CPU. Items were run head against merge base where noted.
- Reported scenario, base vs head: pass. Base: after the timeout, watches rose to 3,750 before teardown. Head: the count never went above 16, and nothing rose after the timeout.
- Retry cadence under sustained load: pass. The six timeout lines were 46 s, 77 s, 137 s, 256 s, and 496 s apart. That is the 30/60/120/240/480 s ladder plus the 10 s deadline and about 6 s of lag. The base caps at 300 s.
- Degraded polling keeps Changes fresh while setups time out: pass. A file change appeared with no watcher running.
- Recovery after the load lifts: pass. The watcher recovered with 4,537 watches, and an edit appeared in under 1 s.
- Callers that reach the user: pass. With no load, create, edit, and delete each showed in the Changes view in about 340 ms. A file made by a real Codex session also appeared in Changes.
- Shutdown or archive while a setup is pending: pass.
- SIGTERM about 5 s into a pending setup, on head: graceful exit in 4.3 s, no forced kill, no leftover processes or watches.
- Same sequence on the merge base, two runs: the worker was force-killed both times, after 17 s and 11 s.
- Archive about 5 s into a pending setup, on head: no retry afterwards, no leftover watches, no polling.
- Regression tests red without the fix, green with it: pass. The head passes 134/134. With the base sources, 9 fail. The file-observer suite passes 32/32 and the server typecheck is clean.
- Native-recursive backend with real
fs.watch, close during the stat phase: pass. The base left 72,071 inotify watches and the process hung. The head left 0.
Outside the checklist
- Archive during a pending setup still logs a timeout line. It printed about 5 s after the archive. Archive does not cancel a setup that is already in flight, on main or here. With this branch, that setup is now aborted at its deadline and never retried, so the cost is bounded at 10 s. This is unchanged from main and has no user-visible effect, so I considered it and did not count it.
project createtimed out once. It hit a 30 sgit statustimeout on a freshly copied repo under deliberate CPU starvation, and the retry succeeded. The PR does not touch that path, and the failure came from the test load. Considered, not counted.- The Files tab tree did not update live. That listing has no live subscription, and the PR does not touch it. Considered, not counted.
Review threads
The one thread (an abort test that passed without the listener) was settled by 3c48abb and is resolved. Item 7 confirmed the red/green.
Reviewed commit: 3c48abb
…utes (getpaseo#9828) * fix(server): abort timed-out watcher setups and back off their retries A watcher subscription that missed WORKSPACE_GIT_WATCHER_SUBSCRIBE_TIMEOUT_MS was abandoned but its startup scan kept walking the tree to completion, and recovery retried every five minutes forever, repeating the full scan each time. On a CPU-starved host that kept dozens of directories rescanning. - FileObserver.subscribe accepts an abort signal that stops the startup scan and rejects; WorkspaceGit aborts setups at the deadline and on dispose. - Consecutive setup timeouts extend the recovery delay past the 5-minute ladder cap, doubling up to one hour; an accepted subscription resets it. Degraded polling keeps its 5-60 s cadence meanwhile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(file-observer): keep an aborted observation's backend from starting Aborting a subscription closes its observation, which can happen while start() is still reading the root. The backend then started after its close had finished, and the native backend opened a root watcher that nothing would close. Co-authored-by: Spencer Shulem <1703265+Spshulem@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(file-observer): abort a subscription mid-scan in the abort test The test aborted before the backend started, so it passed with the abort listener removed. A tree past the Linux directory limit makes a scan that keeps walking fail on the limit instead of rejecting with the reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Spencer Shulem <spshulem@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: paseo-bot[bot] <266920839+paseo-bot[bot]@users.noreply.github.com> Co-authored-by: Spencer Shulem <1703265+Spshulem@users.noreply.github.com> (cherry picked from commit 3325974)
Linked issue
Closes #9822
Refs #9611 (a macOS workspace whose watcher fails on the 250,000-entry limit and rescans every ~5 minutes; that failure is not a setup timeout, so the timeout backoff here does not cover it)
Type of change
Reasoning
On a busy host, a large repo's working-tree or repository-metadata watcher can take longer than the 10 second setup deadline. The daemon gave up waiting at the deadline but let the startup scan keep walking the whole tree, attaching a watcher per directory, and then tore it down. Recovery retried on a ladder capped at 5 minutes, so every timed-out directory ran another full scan every 5 minutes for as long as the host stayed busy, which kept the daemon's main thread stalled and made switching chats slow.
Before and after, on a workspace whose watcher setup misses the deadline:
This builds on @Spshulem's commit from #9818, which found the cause and made both changes, and adds one fix: an abort that landed while the observation was still reading the root closed it before the backend started, and the backend then started anyway. On macOS and Windows that left the root watcher open with nothing to close it.
Goals
Non-goals
QA
file-observer/index.test.tsnow aborts 10 ms into setup on a tree past the Linux backend's 5,000-directory limit. With the abort listener removed it fails on the directory-limit error (2/2 runs); with it, it passes (3/3).file-observer/internal/observation.test.ts: closing an observation before its native backend starts leaves no watcher open. It fails without the guard inObservation.start()(one watcher left open) and passes with it.file-observerandworkspace-git-servicesuites: 228/228. Other server suites that use the file observer or WorkspaceGit (checkout diff, diff subscriptions, worktree session, workspace runtime store, auto-archive, MCP server, project suggestions, session workspaces): 384 passed, 4 skipped.Risk
FileObserver.subscribegains an optionalsignal; callers that don't pass one behave as before.Checklist
npm run typecheckpassesnpm run lintpassesnpm run formatpassesSupersedes
🤖 Generated with Claude Code