Skip to content

fix(fc): reflink-copy base rootfs image instead of full byte copy - #1076

Merged
cdalar merged 5 commits into
mainfrom
fix/fc-rootfs-reflink-copy
Sep 3, 2026
Merged

cdalar merged 5 commits into
mainfrom
fix/fc-rootfs-reflink-copy

Conversation

@cdalar

@cdalar cdalar commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Summary

  • onctl up -n <name> for the fc provider was spending most of its wall time (~4.6s sys time observed) on a plain io.Copy of the entire base rootfs image on every Deploy.
  • The cache disk preparer already avoided this via a copy-on-write reflink (cp --reflink=always); the rootfs preparer now does the same, falling back to a full byte-for-byte copy on filesystems that don't support reflinks.

Test plan

  • go build ./...
  • go vet ./...
  • go test ./internal/providerfc/...
  • gofmt -l clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01PVpF4aLeq6xpokgHiBCy4v

DebugfsRootfsPreparer.Prepare did a plain io.Copy of the entire base
rootfs image on every Deploy, dominating `onctl up` time for the fc
provider (multi-second sys time copying a large image). The cache
disk preparer already avoided this cost via a CoW reflink; the rootfs
path now does the same, falling back to a full copy on filesystems
without reflink support.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVpF4aLeq6xpokgHiBCy4v

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7e7000483c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/providerfc/common.go Resolved
cp --reflink preserves the base image's own mode (typically
world/group-readable), but the destination is a per-VM writable
rootfs holding guest secrets under the shared StateDir. Chmod it to
0600 after a successful reflink, matching what copyFile's plain-copy
fallback already did.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVpF4aLeq6xpokgHiBCy4v

cdalar commented Sep 2, 2026

Copy link
Copy Markdown
Owner Author

The claude-review check is failing on an infra step unrelated to this PR's diff: the action fails while adding the anthropics/claude-code marketplace with HTTPS authentication failed... fatal: could not read Username for 'https://github.com': terminal prompts disabled. This has happened identically on both commits so far (7e70004 and e39aed4), before any review logic runs — it's a credential-helper/auth problem in the workflow's marketplace-setup step, not something this PR's code touches or can fix. Re-running the job once to rule out a one-off; if it fails the same way again, the workflow config needs attention.


Generated by Claude Code

cdalar commented Sep 2, 2026

Copy link
Copy Markdown
Owner Author

Build (stable) failed on TestActionCmd_DownloadFile_404 (cmd/coverage_test.go), which hits the external service https://httpstat.us/404 and got a 502 Bad Gateway instead of the expected 404. That's an external-service hiccup in a pre-existing test unrelated to this PR's change (internal/providerfc/common.go). Re-running the failed job once to confirm.


Generated by Claude Code

cdalar commented Sep 2, 2026

Copy link
Copy Markdown
Owner Author

Build (stable) failed again on the same test: TestActionCmd_DownloadFile_404 (cmd/coverage_test.go:153) got a 502 Bad Gateway from https://httpstat.us/404 instead of the expected 404, on both this run and the previous one. That confirms it's not a one-off flake — httpstat.us is consistently unreliable from this runner — but it's still unrelated to this PR's change (internal/providerfc/common.go), which the CI graph confirms: Lint, Vuln, security, CodeQL, Analyze(go), and Test deployment are all green on this same commit.

Root cause and fix are outside this PR's scope (pre-existing test, unrelated file), so per the drive-to-green policy I'm not widening this PR to patch it. For whoever picks this up: swap the external call for an httptest.NewServer returning a fixed 404, e.g.:

srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
err := downloadFile(srv.URL, tempFile)

That removes the external dependency entirely rather than just re-pointing it. I won't re-run this again (already used one re-run to rule out a transient flake); merging is blocked on this until it's fixed or the branch protection rule for Build (stable) is relaxed to tolerate this known-unrelated failure.


Generated by Claude Code

Measured against a real firecracker microVM: a failed dial while the
guest is still booting fails in well under a millisecond once the OS
gives up on it, so the 500ms sleep between attempts was pure
detection-latency slop on top of actual guest boot time. Only affects
the fc provider today (WaitForSSH's only caller); a slower/higher-
latency SSH target just sees more frequent harmless retries, not a
correctness risk.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PVpF4aLeq6xpokgHiBCy4v
@cdalar

cdalar commented Sep 3, 2026

Copy link
Copy Markdown
Owner Author

Verified in production

Deployed and tested on both live fc-hosts (aimax, fc-host):

  • Timing: onctl up for the fc provider dropped from real 8.371s / sys 4.670s to real ~1.8s / sys ~0.02-0.09s — a ~4.6x wall-clock improvement, entirely from the rootfs reflink fix.
  • Reflink confirmed real (not just faster disk): df showed zero additional physical disk usage after creating a 20GB rootfs image — proof it's true copy-on-write, not a full copy.
  • Permissions confirmed correct: reflinked rootfs images come out 0600 as intended (verified via stat).
  • Rolled out to both hosts without disrupting live traffic (in-flight GitHub Actions runners and a terminal-rental VM were left untouched during the binary swap).

Investigated further to shave the remaining ~1.8s (guest boot + SSH readiness) via SSH poll tuning, ARP pre-seeding, console-log-based readiness detection, and systemd unit reordering on a disposable clone of the golden image — all tested empirically, all had no measurable benefit beyond what's in this PR (the guest's own boot time and sshd's daemon-init cost are the real floor). Only the harmless poll-interval tightening (500ms→100ms) made it into this PR; everything else was reverted after testing showed no gain.

Ready to merge.

cdalar commented Sep 3, 2026

Copy link
Copy Markdown
Owner Author

claude-review failed again on the new commit (f3a871d) with the same pre-existing infra issue: the action fails adding the anthropics/claude-code marketplace (HTTPS authentication failed), before any review logic runs — unrelated to this PR's code (now including the new WaitForSSH poll-interval change). Re-running once.


Generated by Claude Code

@cdalar cdalar mentioned this pull request Sep 3, 2026
1 task done
govulncheck flagged GO-2026-6355 and GO-2026-6354 (DoS on deadlocked
ssh channels) reachable from tools.Remote.NewSSHConnection's ssh.Dial
call in internal/tools/remote-run.go, the file this PR's WaitForSSH
change touches. Both are fixed upstream in x/crypto v0.56.0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fsg3yjczdwMHsefNVYPd6J
@socket-security

socket-security Bot commented Sep 3, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Updatedgolang/​golang.org/​x/​crypto@​v0.55.0 ⏵ v0.56.074 +1100100100100

View full report

This test hit https://httpstat.us/404 directly and had failed 4
consecutive CI runs on this PR with 502 Bad Gateway from that service,
unrelated to any of this PR's own changes but blocking merge since
Build (stable) is a required check. Serve the 404 from a local
httptest.Server instead, removing the external dependency entirely.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fsg3yjczdwMHsefNVYPd6J

cdalar commented Sep 3, 2026

Copy link
Copy Markdown
Owner Author

Pushed db50fd7: swapped TestActionCmd_DownloadFile_404's dependency on https://httpstat.us/404 for a local httptest.Server, since that external service has now returned 502 on 4 consecutive Build (stable) runs on this PR and was blocking merge on an otherwise unrelated flake. Verified locally — the test passes reliably now, and go vet/go build are clean. This was the last remaining blocker; everything else on this PR (Lint, Vuln, security, CodeQL, Analyze, Test deployment, claude-review) is green.


Generated by Claude Code

cdalar added a commit that referenced this pull request Sep 3, 2026
Fixes GO-2026-6355 and GO-2026-6354, both DoS-via-deadlock issues in
golang.org/x/crypto/ssh reachable through tools.Remote.NewSSHConnection's
ssh.Dial call — the govulncheck CI job was failing on this across
unrelated PRs (see #1076).


Claude-Session: https://claude.ai/code/session_01PVpF4aLeq6xpokgHiBCy4v

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
@cdalar
cdalar merged commit 86a1141 into main Sep 3, 2026
13 of 14 checks passed
@cdalar
cdalar deleted the fix/fc-rootfs-reflink-copy branch September 3, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ 换网址