Skip to content

fix(fc): reconcile stale running/paused status against actual process liveness - #954

Merged
cdalar merged 2 commits into
mainfrom
fix/fc-reconcile-stale-status
Jul 8, 2026
Merged

cdalar merged 2 commits into
mainfrom
fix/fc-reconcile-stale-status

Conversation

@cdalar

@cdalar cdalar commented Jul 8, 2026

Copy link
Copy Markdown
Owner

Summary

  • `onctl -p fc ls` kept reporting Firecracker microVMs as `running` indefinitely after their firecracker process died out-of-band — the case that surfaced this: rebooting the host for a kernel upgrade. Every firecracker VMM process dies on host reboot, but nothing updates `metadata.json`'s persisted `status`, so `List()` just replayed the stale value forever.
  • `Deploy()` had the same blind spot: it treated any existing `metadata.json` as proof the microVM was already up and returned it unchanged — so re-running `onctl create` for a dead microVM silently no-op'd instead of recreating it.
  • `Destroy()` already guarded against this class of bug (checking `Process.IsRunning` + `Process.Owns` before signaling a persisted PID, to avoid killing an unrelated process that reused the PID after reboot) — this PR gives `List`/`Deploy`/`GetByName` the same treatment via a shared `loadAndReconcile` helper.

Changes

  • New `fcStatusStopped` status, and `isAlive`/`loadAndReconcile` helpers in `pkg/cloud/fc.go`.
  • `listAll()` (used by `List`, `ListPaused`, `usedIPs`) now reconciles each record's status against actual process liveness, self-healing a stale `running`/`paused` status to `stopped` on disk.
  • `Deploy()` redeploys over a `stopped` record (cleans up the stale tap device + state dir) instead of returning it unchanged.
  • `GetByName()` reconciles too, so `SSHInto` sees an accurate status and now fails fast with a clear message instead of hanging against a dead socket.

Test plan

  • `go build ./...`
  • `go vet ./...`
  • `golangci-lint run ./pkg/cloud/...` — 0 issues
  • `go test ./pkg/cloud/...` — all passing, including 3 new tests:
    • `TestProviderFC_List_ReconcilesDeadProcess`
    • `TestProviderFC_Deploy_RecreatesStaleRecord`
    • `TestProviderFC_GetByName_ReconcilesDeadProcess`
  • Existing `TestProviderFC_Deploy_Idempotent` and `TestProviderFC_PauseResume` still pass unchanged, confirming reconciliation is a no-op when the process is genuinely alive.

… liveness

onctl -p fc ls kept reporting microVMs as "running" indefinitely after
their firecracker process died out-of-band — most notably a host reboot,
which every firecracker VMM dies to but leaves metadata.json's
last-written status untouched. Deploy() had the same blind spot: it
treated a stale "running" record as proof the microVM was already up
and silently no-op'd instead of recreating it.

List/Deploy/GetByName now reconcile the persisted status against
Process.IsRunning + Process.Owns (the same liveness check Destroy
already used to guard against PID reuse), self-healing a stale record
to a new "stopped" status on disk. Deploy() redeploys over a stopped
record instead of returning it unchanged, and SSHInto now fails fast
with a clear message instead of hanging against a dead socket.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 527d04a34e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/cloud/fc.go Resolved
onctl create checks provider.List() for a same-named VM before calling
Deploy, but List() includes fc records reconciled to "stopped" (dead
process, e.g. after a host reboot). That made the check abort with "VM
exists" instead of ever reaching Deploy's stale-record recreate path,
defeating the reconciliation added in the previous commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RuNxaYWJi7HTNLUAZjPC1y
@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 62.96296% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 24.12%. Comparing base (cc7aec1) to head (2bf29e1).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
pkg/cloud/fc.go 68.00% 5 Missing and 3 partials ⚠️
cmd/create.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #954      +/-   ##
==========================================
+ Coverage   23.94%   24.12%   +0.17%     
==========================================
  Files          44       44              
  Lines        4418     4440      +22     
==========================================
+ Hits         1058     1071      +13     
- Misses       3229     3235       +6     
- Partials      131      134       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cdalar
cdalar merged commit 9ba7d86 into main Jul 8, 2026
17 checks passed
cdalar added a commit that referenced this pull request Jul 8, 2026
#955)

PR #954's reconciliation persisted a dead fc microVM's status as
"stopped" — but cmd/list.go's isPausedStatus() already treats the
literal string "stopped" as the cross-provider paused/resumable
convention (AWS "stopped", GCP "TERMINATED", Azure "deallocated") and
folds it into onctl ls's PAUSED section. That collision made a
reboot-killed microVM (no restart-from-disk, must be destroyed and
recreated) render as merely "paused" (implying it can be resumed),
which is exactly what this fix was supposed to stop lying about.

Renames fcStatusStopped to fcStatusDead ("dead") throughout pkg/cloud
and the cmd/create.go duplicate-name check, and adds
TestIsPausedStatus in cmd/list_test.go to lock in the distinction.
@cdalar
cdalar deleted the fix/fc-reconcile-stale-status branch August 19, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ 换网址