Skip to content

fc/ch: drop the stale neighbour entry for a reused guest IP - #1190

Merged
cdalar merged 1 commit into
mainfrom
ai/fc-stale-neighbour
Oct 12, 2026
Merged

cdalar merged 1 commit into
mainfrom
ai/fc-stale-neighbour

Conversation

@cdalar

@cdalar cdalar commented Oct 12, 2026

Copy link
Copy Markdown
Owner

Problem

onctl create on a Firecracker host can spend 8 seconds in "Waiting for SSH" for a guest that has been listening since about 1.2 s.

Deploy allocates the lowest free address on the bridge, but each microVM's MAC is derived from its name. A reused address therefore arrives with a new MAC, while the host's neighbour cache may still map it to the MAC of a microVM that no longer exists. The kernel trusts the STALE entry, sends to the dead MAC, and re-resolves only after delay_first_probe_time (5 s) plus ucast_solicit unicast probes (3 x 1 s).

It only shows on hosts whose bridge stays up between creates (for example when long-lived VMs share it). A bridge that loses carrier when its last VM goes away flushes the cache, which is why one host was fast and another slow on the same guest image.

Measurements (same host, same image, 2 vCPU / 8 GB)

onctl create guest sshd listening
stale entry present 8.60 s 1.16 s
after ip neigh flush dev fcbr0 nud stale 1.55 s 1.14 s

Change

  • NetworkManager gains ForgetNeighbor(bridge, ip); the Linux implementations run ip neigh del <ip> dev <bridge> and treat a missing entry as success.
  • ProviderFC.Deploy and ProviderCH.Deploy call it for the address they just allocated, before the guest boots. The ch provider shares the allocator and the name-derived MAC, so it has the same exposure.
  • Best-effort: a failure is logged at debug level and costs only the old delay.

Resume and Restart are untouched: they keep the VM's existing IP and MAC.

Test plan

  • go build ./..., go vet (GOOS=linux) on the touched packages
  • go test ./pkg/cloud/ ./internal/providerfc/ ./internal/providerch/ — deploy tests for both providers now assert the call
  • golangci-lint run on the touched packages: 0 issues
  • Not yet run on a real host: the 1.55 s figure comes from flushing the entry by hand, which is the state this change produces, not from a build of this branch

🤖 Generated with Claude Code

https://claude.ai/code/session_01AKmJku7417CUEaGfeQRx8C

Deploy hands out the lowest free address on the bridge but derives each
microVM's MAC from its name, so a reused address comes with a new MAC.
The host's neighbour cache can still map that address to the MAC of a
microVM that is gone. The kernel trusts a STALE entry, sends to the dead
MAC, and only re-resolves after delay_first_probe_time (5s) plus three
unicast probes (1s each): 8s in which the guest is up and listening but
unreachable from the host, all of it spent in WaitForSSH.

Measured on a pool host whose bridge never goes down (long-lived VMs
share it, so the cache is never flushed): `onctl create` took 8.6s with
guest sshd listening at 1.2s; after `ip neigh flush` of the stale
entries the same create took 1.55s. A host whose bridge loses carrier
between VMs flushes the cache by accident and never showed it.

Deploy now deletes the neighbour entry for the address it just
allocated, before the guest boots. Best-effort: a failure is logged and
only costs the old delay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKmJku7417CUEaGfeQRx8C
@cdalar
cdalar enabled auto-merge (squash) October 12, 2026 19:21
@cdalar
cdalar merged commit eeea936 into main Oct 12, 2026
12 checks passed
@cdalar
cdalar deleted the ai/fc-stale-neighbour branch October 12, 2026 19:25
@codecov

codecov Bot commented Oct 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 16.66667% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 39.79%. Comparing base (1ddff76) to head (b20ab20).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
internal/providerch/common.go 0.00% 4 Missing ⚠️
internal/providerfc/common.go 0.00% 4 Missing ⚠️
pkg/cloud/ch.go 50.00% 1 Missing ⚠️
pkg/cloud/fc.go 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1190      +/-   ##
==========================================
- Coverage   39.83%   39.79%   -0.04%     
==========================================
  Files          67       67              
  Lines        7298     7310      +12     
==========================================
+ Hits         2907     2909       +2     
- Misses       4391     4401      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址