Confirm conntrack state before auto-standby - #319
Open
sjmiller609 wants to merge 6 commits into
Open
Conversation
The idle countdown is driven entirely by the conntrack event stream, so a dropped NEW event (receive buffer overrun, or the reconnect gap after a stream error) leaves activeInbound empty while the guest still holds a live connection. Nothing re-read the conntrack table between the countdown starting and the timer firing, so an instance could enter standby with a connection open; the only correction was the 5 minute snapshot sync, which never runs in time for a shorter idle timeout. Dump the conntrack table in handleStandbyTimer and abort the attempt when the instance still has matching inbound connections, treating a failed dump as busy. Also resync on stream restore so state does not stay stale for a full snapshot interval after a reconnect.
sjmiller609
marked this pull request as ready for review
July 28, 2026 15:44
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
The confirmation added in the previous commit changed in-memory state without writing it back, so a restart could resume a stale idle_since and standby with no fresh countdown. Persist the runtime on both confirmation outcomes. Mark the observer connected after the post-restore resync instead of before: a failed conntrack dump inside the resync records an observer error, which otherwise left a live subscription reporting StatusError until the next reconnect.
Activity can be processed between the idle timer firing and the confirmation running, and it already owns idleSince and the reconcile loop. Bail out in that case instead of overwriting the active state with a fresh idle countdown when the conntrack dump fails.
The confirmation dump releases the lock, so a standby worker on its failure path can arm a replacement idle timer in that window. Assigning state.timer = nil orphaned that AfterFunc, leaving a stray firing queued behind the standby that was about to run. Cancel the timer instead, which stops it and clears nextStandbyAt the same way.
…standby Conflict in handleStandbyTimer: main added a stale-delivery check that re-arms when the current deadline has not elapsed, and this branch replaced the timer teardown with cancelTimerLocked. Kept both, with the stale check ahead of the conntrack confirmation so a held or re-armed instance is not dumped or given a restarted countdown, and the confirmation only runs once the deadline is real.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5816ec3. Configure here.
The confirmation releases the lock, so a hold placed while it runs extends the deadline without the standby path noticing, putting the instance into standby in spite of the hold. Check the deadline on both sides of the confirmation via standbyDeadlineElapsedLocked, which also replaces the spent timer.
hiroTamada
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The auto-standby idle countdown is driven entirely by the conntrack event stream.
state.activeInboundis built from NEW/DESTROY events, and once it drains the countdown timer arms — after that nothing re-reads the conntrack table before the timer fires. So a single dropped NEW event (receive buffer overrun, or the reconnect gap after a stream error, where the restored subscription previously did no resync) is enough to standby an instance that still has a live connection. The only correction wasperiodicSnapshotSyncat 5 minutes, which never runs in time for a shorteridle_timeout.Two changes, both in
lib/autostandby/controller.go:handleStandbyTimernow dumps the conntrack table and classifies it with the instance's policy before committing to standby. If matching inbound connections are still present, it repopulatesactiveInbound, clearsidleSince, arms the reconcile timer, and skips the attempt. If the dump or classification fails, the instance is treated as busy and the countdown restarts rather than risking standby on an unverified instance. One dump per standby attempt, on the same event-loop goroutine that already dumps every 2s viahandleActiveReconcile.streamReadycase inRuncallsstartupResyncafter a subscription is restored, so state is rebuilt from the table immediately instead of staying stale for up to a full snapshot interval.Tests
go test ./lib/autostandby/ -racepasses, including three new cases: standby skipped when conntrack reports a connection the event stream missed, countdown re-armed when the confirmation dump fails, and aRun-level test that fails the stream and asserts the instance flips back to active after the reconnect resync../lib/instancesand./cmd/api/apido not build in this checkout (missing embeddedcloud-hypervisor/caddy/guest-agentbinaries, unrelated to this change), so their auto-standby tests were not run.Note
Medium Risk
Changes when VMs enter standby (snapshot IO) based on authoritative conntrack reads; behavior is conservative on errors but affects production instance lifecycle timing.
Overview
Fixes a case where auto-standby could snapshot a VM that still had live inbound traffic when conntrack NEW events were missed (stream gaps or buffer overruns) and idle state was driven only by the event stream.
Before standby,
handleStandbyTimernow runsconfirmIdleBeforeStandby: a conntrack table dump and policy match. If connections remain, it restores active state and skips standby; if the dump fails, it restarts the idle countdown instead of standing by on unverified idle. Deadline checks are factored intostandbyDeadlineElapsedLocked, with a second pass after confirmation for holds/re-arms that happen while the lock was released.On conntrack subscription restore,
RuncallsstartupResyncbefore marking the observer connected, so state is reseeded from the table immediately rather than staying stale until periodic sync; observer “connected” is set after resync so a failed dump does not look like a dead subscription.Tests cover missed events, failed confirmation, races (hold/re-arm during confirmation), and
Run-level stream restore behavior.Reviewed by Cursor Bugbot for commit 571b961. Bugbot is set up for automated code reviews on this repo. Configure here.