Skip to content

Commit 3325974

Browse files
paseo-bot[bot]Spshulemclaude
authored
Stop timed-out watcher setups from rescanning large repos every 5 minutes (#9828)
* fix(server): abort timed-out watcher setups and back off their retries A watcher subscription that missed WORKSPACE_GIT_WATCHER_SUBSCRIBE_TIMEOUT_MS was abandoned but its startup scan kept walking the tree to completion, and recovery retried every five minutes forever, repeating the full scan each time. On a CPU-starved host that kept dozens of directories rescanning. - FileObserver.subscribe accepts an abort signal that stops the startup scan and rejects; WorkspaceGit aborts setups at the deadline and on dispose. - Consecutive setup timeouts extend the recovery delay past the 5-minute ladder cap, doubling up to one hour; an accepted subscription resets it. Degraded polling keeps its 5-60 s cadence meanwhile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(file-observer): keep an aborted observation's backend from starting Aborting a subscription closes its observation, which can happen while start() is still reading the root. The backend then started after its close had finished, and the native backend opened a root watcher that nothing would close. Co-authored-by: Spencer Shulem <1703265+Spshulem@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(file-observer): abort a subscription mid-scan in the abort test The test aborted before the backend started, so it passed with the abort listener removed. A tree past the Linux directory limit makes a scan that keeps walking fail on the limit instead of rejecting with the reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Spencer Shulem <spshulem@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: paseo-bot[bot] <266920839+paseo-bot[bot]@users.noreply.github.com> Co-authored-by: Spencer Shulem <1703265+Spshulem@users.noreply.github.com>
1 parent 41c294c commit 3325974

8 files changed

Lines changed: 251 additions & 7 deletions

File tree

‎docs/file-observation.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Do not replace the Linux implementation with `fs.watch({ recursive: true })`. No
66

77
Closing an observation must remain safe after its root is renamed or removed. Workspace archive removes owned worktrees while releasing observation references, so teardown cannot assume the watched path still exists.
88

9-
`unsubscribe()` is an awaited barrier. Once it resolves, no scan, reconciliation, queued event, native handle, or callback may remain. `updateIgnore()` is the equivalent barrier for excluded paths: events queued below a newly excluded root cannot be delivered after it resolves. Treat path disappearance while scanning, attaching, or receiving a watcher error as rename/archive churn. Surface permission and resource errors so WorkspaceGit can enter bounded polling.
9+
`unsubscribe()` is an awaited barrier. Once it resolves, no scan, reconciliation, queued event, native handle, or callback may remain. `updateIgnore()` is the equivalent barrier for excluded paths: events queued below a newly excluded root cannot be delivered after it resolves. Treat path disappearance while scanning, attaching, or receiving a watcher error as rename/archive churn. Surface permission and resource errors so WorkspaceGit can enter bounded polling. `subscribe()` accepts an abort signal; aborting it before it resolves stops the startup scan and rejects. WorkspaceGit aborts every subscription that misses its setup deadline, because an abandoned scan otherwise keeps walking the tree. A timed-out directory retries on the recovery ladder (30 seconds doubling to five minutes), and each consecutive timeout keeps doubling the delay up to one hour. Under sustained CPU load the next scan misses the deadline too, and each attempt repeats the full scan. An accepted subscription resets the timeout streak. Degraded polling keeps its own cadence throughout.
1010

1111
Linux topology repair is trailing-debounced and scoped to the renamed path. A file rename needs one classification stat; a new or moved directory scans only its subtree. Full-tree scans are reserved for startup and ignore-set replacement. A subscription reports changes after `subscribe()` resolves, not the files that already existed. macOS FSEvents can still replay writes made moments before the stream started. Consumers read their initial state after `subscribe()` resolves; WorkspaceGit re-reads its checkout once observation setup completes, because its first refresh runs in parallel with setup. The Linux startup scan therefore emits nothing; only a scoped reconciliation of new topology reports files found before their directory's watcher was installed. Startup creates made every cold diff compute and discard a first revision. Benchmark startup cost with pre-populated trees.
1212

‎packages/server/src/server/file-observer/index.test.ts‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -327,6 +327,39 @@ test("unsubscribe cancels reconciliation queued by a write burst", async () => {
327327
expect(callback).toHaveBeenCalledTimes(callsAtClose);
328328
}, 20_000);
329329

330+
test("aborting a pending subscription stops its startup scan and rejects", async () => {
331+
const root = await createRoot();
332+
// More directories than the Linux backend watches: a scan that keeps walking after the abort
333+
// fails on the directory limit instead of rejecting with the abort reason.
334+
await Promise.all(
335+
Array.from({ length: 80 }, (_, outer) =>
336+
Promise.all(
337+
Array.from({ length: 70 }, (__, inner) =>
338+
mkdir(join(root, `dir-${outer}`, `nested-${inner}`), { recursive: true }),
339+
),
340+
),
341+
),
342+
);
343+
const observer = createObserver();
344+
const controller = new AbortController();
345+
const reason = new Error("setup deadline passed");
346+
const callback = vi.fn();
347+
348+
const subscription = observer.subscribe(root, callback, { signal: controller.signal });
349+
await new Promise((resolve) => setTimeout(resolve, 10));
350+
controller.abort(reason);
351+
352+
await expect(subscription).rejects.toBe(reason);
353+
expect(observer.getDiagnostics()).toMatchObject({
354+
activeObservationCount: 0,
355+
nativeHandleCount: 0,
356+
pendingReconciliationWorkCount: 0,
357+
});
358+
await writeFile(join(root, "dir-0", "after-abort.txt"), "late");
359+
await new Promise((resolve) => setTimeout(resolve, 100));
360+
expect(callback).not.toHaveBeenCalled();
361+
}, 20_000);
362+
330363
test("aggregate diagnostics return to their lifecycle baseline", async () => {
331364
const root = await createRoot();
332365
const observer = createObserver();

‎packages/server/src/server/file-observer/index.ts‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ export interface FileChange {
1414

1515
export interface FileObserverOptions {
1616
ignore?: string[];
17+
// Aborting before `subscribe()` resolves stops the startup scan and rejects with the reason.
18+
signal?: AbortSignal;
1719
}
1820

1921
export type FileObserverCallback = (error: Error | null, events: FileChange[]) => void;
@@ -105,7 +107,19 @@ class FileObserverService implements FileObserver {
105107
createBackend: this.createBackend,
106108
onClosed: () => state.observations.delete(observation),
107109
});
108-
await observation.start();
110+
const { signal } = options;
111+
signal?.throwIfAborted();
112+
const abort = () => void observation.close();
113+
signal?.addEventListener("abort", abort, { once: true });
114+
try {
115+
await observation.start();
116+
} finally {
117+
signal?.removeEventListener("abort", abort);
118+
}
119+
if (signal?.aborted) {
120+
await observation.close();
121+
throw signal.reason;
122+
}
109123
if (this.closePromise) {
110124
await observation.close();
111125
throw new Error("File observer is closed");
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
import { mkdtemp, rm } from "node:fs/promises";
2+
import { tmpdir } from "node:os";
3+
import { join } from "node:path";
4+
import { expect, test } from "vitest";
5+
import { createFileObserver } from "../index.js";
6+
import { createNativeRecursiveBackend } from "./native-recursive.js";
7+
import { createManagedObservation } from "./observation.js";
8+
import { createObserverPaths } from "./paths.js";
9+
10+
// Aborting a subscription closes its observation while start() may still be
11+
// reading the root, before the backend has started.
12+
test("closing an observation before its backend starts leaves no native watcher open", async () => {
13+
const root = await mkdtemp(join(tmpdir(), "observation-close-"));
14+
const paths = createObserverPaths(process.platform);
15+
let openWatchers = 0;
16+
const observation = createManagedObservation({
17+
root,
18+
callback: () => {},
19+
options: {},
20+
paths,
21+
metrics: createFileObserver().getDiagnostics(),
22+
createBackend: (host) =>
23+
createNativeRecursiveBackend(host, paths, () => {
24+
openWatchers += 1;
25+
let open = true;
26+
return {
27+
close: () => {
28+
if (open) openWatchers -= 1;
29+
open = false;
30+
},
31+
on: () => {},
32+
};
33+
}),
34+
onClosed: () => {},
35+
});
36+
try {
37+
const started = observation.start();
38+
const closed = observation.close();
39+
await started;
40+
await closed;
41+
42+
expect(openWatchers).toBe(0);
43+
} finally {
44+
await rm(root, { recursive: true, force: true });
45+
}
46+
});

‎packages/server/src/server/file-observer/internal/observation.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,9 @@ class Observation implements ManagedObservation, ObservationHost {
6969
if (!rootStats.isDirectory()) {
7070
throw new Error(`File observation requires a directory: ${this.root}`);
7171
}
72+
// An aborted subscription can close the observation while the root is being read; the backend
73+
// must not start after its close has already finished.
74+
if (!this.isActive()) return;
7275
try {
7376
await this.backend.start();
7477
} catch (error) {

‎packages/server/src/server/workspace-git-service.primitive.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2195,6 +2195,7 @@ describe("WorkspaceGitServiceImpl D2 read methods", () => {
21952195
expect(subscribe).toHaveBeenCalledTimes(1);
21962196
expect(subscribe).toHaveBeenCalledWith(repoDir, expect.any(Function), {
21972197
ignore: [join(repoDir, ".git"), ignoredRoot],
2198+
signal: expect.any(AbortSignal),
21982199
});
21992200

22002201
subscription.unsubscribe();

‎packages/server/src/server/workspace-git-service.test.ts‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -477,6 +477,56 @@ async function advanceThroughWorkingTreeRecoveryAttempts(
477477
}
478478
}
479479

480+
// Stalls every subscription for `watchPath` the way a CPU-starved startup scan does, until the
481+
// service aborts it at the subscribe deadline. Other paths subscribe immediately.
482+
function createStallingWatchSetup(watchPath: string) {
483+
const signals: AbortSignal[] = [];
484+
let acceptNext = false;
485+
let acceptedCallback: WatchCallback | null = null;
486+
const subscribe = vi.fn(
487+
async (directory: string, callback: WatchCallback, options?: { signal?: AbortSignal }) => {
488+
if (directory !== watchPath) {
489+
return createAsyncSubscription();
490+
}
491+
const signal = options?.signal ?? new AbortController().signal;
492+
signals.push(signal);
493+
if (acceptNext) {
494+
acceptNext = false;
495+
acceptedCallback = callback;
496+
return createAsyncSubscription();
497+
}
498+
return new Promise<ReturnType<typeof createAsyncSubscription>>((_resolve, reject) => {
499+
signal.addEventListener("abort", () => reject(signal.reason), { once: true });
500+
});
501+
},
502+
);
503+
return {
504+
subscribe,
505+
signals,
506+
acceptNextAttempt() {
507+
acceptNext = true;
508+
},
509+
failAcceptedWatcher() {
510+
acceptedCallback?.(new Error("accepted watcher stopped"), []);
511+
},
512+
async expectAttemptAbortedAtDeadline() {
513+
const signal = signals.at(-1);
514+
expect(signal?.aborted).toBe(false);
515+
await vi.advanceTimersByTimeAsync(WORKSPACE_GIT_WATCHER_SUBSCRIBE_TIMEOUT_MS);
516+
expect(signal?.aborted).toBe(true);
517+
expect(signal?.reason).toMatchObject({ name: "WorkspaceGitWatcherSubscriptionTimeoutError" });
518+
},
519+
async expectNextAttemptAfter(delayMs: number, service: WorkspaceGitServiceImpl) {
520+
const attempts = signals.length;
521+
await vi.advanceTimersByTimeAsync(delayMs - 1);
522+
await waitForRefreshesToSettle(service);
523+
expect(signals).toHaveLength(attempts);
524+
await vi.advanceTimersByTimeAsync(1);
525+
await settle(() => expect(signals).toHaveLength(attempts + 1));
526+
},
527+
};
528+
}
529+
480530
describe("WorkspaceGitServiceImpl", () => {
481531
beforeEach(() => {
482532
vi.useFakeTimers();
@@ -1573,6 +1623,7 @@ describe("WorkspaceGitServiceImpl", () => {
15731623
expect(subscribe).toHaveBeenCalledTimes(1);
15741624
expect(subscribe).toHaveBeenCalledWith(REPO_CWD, expect.any(Function), {
15751625
ignore: [join(REPO_CWD, ".git")],
1626+
signal: expect.any(AbortSignal),
15761627
});
15771628

15781629
subscription.unsubscribe();
@@ -1648,6 +1699,67 @@ describe("WorkspaceGitServiceImpl", () => {
16481699
15_000,
16491700
);
16501701

1702+
test.each(["repository metadata", "working tree"])(
1703+
"aborts each timed-out %s watcher setup and backs retries off toward an hour",
1704+
async (kind) => {
1705+
const watchPath = kind === "repository metadata" ? join(REPO_CWD, ".git") : REPO_CWD;
1706+
const setup = createStallingWatchSetup(watchPath);
1707+
const getCheckoutStatus = vi.fn(async (cwd: string) => createCheckoutStatus(cwd));
1708+
const service = createService({ subscribe: setup.subscribe, getCheckoutStatus });
1709+
const subscription = service.registerWorkspace({ cwd: REPO_CWD }, vi.fn());
1710+
try {
1711+
await settle(() => expect(setup.signals).toHaveLength(1));
1712+
await setup.expectAttemptAbortedAtDeadline();
1713+
await waitForObservationToSettle(service, 1);
1714+
for (const delayMs of [
1715+
30_000, 60_000, 120_000, 240_000, 480_000, 960_000, 1_920_000, 3_600_000,
1716+
]) {
1717+
await setup.expectNextAttemptAfter(delayMs, service);
1718+
await setup.expectAttemptAbortedAtDeadline();
1719+
}
1720+
// Degraded polling keeps the checkout fresh while the watcher waits out an hour.
1721+
const readsBeforeCappedWait = getCheckoutStatus.mock.calls.length;
1722+
await setup.expectNextAttemptAfter(3_600_000, service);
1723+
expect(getCheckoutStatus.mock.calls.length).toBeGreaterThanOrEqual(
1724+
readsBeforeCappedWait + 59,
1725+
);
1726+
} finally {
1727+
subscription.unsubscribe();
1728+
await service.dispose();
1729+
}
1730+
},
1731+
30_000,
1732+
);
1733+
1734+
test.each(["repository metadata", "working tree"])(
1735+
"an accepted %s watcher subscription resets the setup-timeout backoff",
1736+
async (kind) => {
1737+
const watchPath = kind === "repository metadata" ? join(REPO_CWD, ".git") : REPO_CWD;
1738+
const setup = createStallingWatchSetup(watchPath);
1739+
const service = createService({ subscribe: setup.subscribe });
1740+
const subscription = service.registerWorkspace({ cwd: REPO_CWD }, vi.fn());
1741+
try {
1742+
await settle(() => expect(setup.signals).toHaveLength(1));
1743+
await setup.expectAttemptAbortedAtDeadline();
1744+
for (const delayMs of [30_000, 60_000, 120_000, 240_000, 480_000]) {
1745+
await setup.expectNextAttemptAfter(delayMs, service);
1746+
await setup.expectAttemptAbortedAtDeadline();
1747+
}
1748+
setup.acceptNextAttempt();
1749+
await setup.expectNextAttemptAfter(960_000, service);
1750+
setup.failAcceptedWatcher();
1751+
// The ordinary recovery ladder's ceiling applies again, not the 32-minute timeout step.
1752+
await setup.expectNextAttemptAfter(300_000, service);
1753+
await setup.expectAttemptAbortedAtDeadline();
1754+
await setup.expectNextAttemptAfter(300_000, service);
1755+
} finally {
1756+
subscription.unsubscribe();
1757+
await service.dispose();
1758+
}
1759+
},
1760+
30_000,
1761+
);
1762+
16511763
test.each(["repository metadata", "working tree"])(
16521764
"ignores expired %s watcher callbacks after recovery",
16531765
async (kind) => {
@@ -1873,6 +1985,7 @@ describe("WorkspaceGitServiceImpl", () => {
18731985
expect(subscription.repoRoot).toBeNull();
18741986
expect(subscribe).toHaveBeenCalledWith(plainCwd, expect.any(Function), {
18751987
ignore: [join(plainCwd, ".git")],
1988+
signal: expect.any(AbortSignal),
18761989
});
18771990
expect(vi.getTimerCount()).toBe(1);
18781991

0 commit comments

Comments
 (0)
⌂ 换网址