Repository navigation
fc/ch: drop the stale neighbour entry for a reused guest IP - #1190
Merged
Merged
Conversation
Deploy hands out the lowest free address on the bridge but derives each microVM's MAC from its name, so a reused address comes with a new MAC. The host's neighbour cache can still map that address to the MAC of a microVM that is gone. The kernel trusts a STALE entry, sends to the dead MAC, and only re-resolves after delay_first_probe_time (5s) plus three unicast probes (1s each): 8s in which the guest is up and listening but unreachable from the host, all of it spent in WaitForSSH. Measured on a pool host whose bridge never goes down (long-lived VMs share it, so the cache is never flushed): `onctl create` took 8.6s with guest sshd listening at 1.2s; after `ip neigh flush` of the stale entries the same create took 1.55s. A host whose bridge loses carrier between VMs flushes the cache by accident and never showed it. Deploy now deletes the neighbour entry for the address it just allocated, before the guest boots. Best-effort: a failure is logged and only costs the old delay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKmJku7417CUEaGfeQRx8C
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1190 +/- ##
==========================================
- Coverage 39.83% 39.79% -0.04%
==========================================
Files 67 67
Lines 7298 7310 +12
==========================================
+ Hits 2907 2909 +2
- Misses 4391 4401 +10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
onctl createon a Firecracker host can spend 8 seconds in "Waiting for SSH" for a guest that has been listening since about 1.2 s.Deploy allocates the lowest free address on the bridge, but each microVM's MAC is derived from its name. A reused address therefore arrives with a new MAC, while the host's neighbour cache may still map it to the MAC of a microVM that no longer exists. The kernel trusts the
STALEentry, sends to the dead MAC, and re-resolves only afterdelay_first_probe_time(5 s) plusucast_solicitunicast probes (3 x 1 s).It only shows on hosts whose bridge stays up between creates (for example when long-lived VMs share it). A bridge that loses carrier when its last VM goes away flushes the cache, which is why one host was fast and another slow on the same guest image.
Measurements (same host, same image, 2 vCPU / 8 GB)
onctl createip neigh flush dev fcbr0 nud staleChange
NetworkManagergainsForgetNeighbor(bridge, ip); the Linux implementations runip neigh del <ip> dev <bridge>and treat a missing entry as success.ProviderFC.DeployandProviderCH.Deploycall it for the address they just allocated, before the guest boots. The ch provider shares the allocator and the name-derived MAC, so it has the same exposure.Resume and Restart are untouched: they keep the VM's existing IP and MAC.
Test plan
go build ./...,go vet(GOOS=linux) on the touched packagesgo test ./pkg/cloud/ ./internal/providerfc/ ./internal/providerch/— deploy tests for both providers now assert the callgolangci-lint runon the touched packages: 0 issues🤖 Generated with Claude Code
https://claude.ai/code/session_01AKmJku7417CUEaGfeQRx8C