Skip to content

Retry transient fresh LXC starts #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gitcommit90 merged 1 commit into from
Jul 29, 2026
+27 −6
Merged

Retry transient fresh LXC starts #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
bootstrap. A channel no longer repeatedly creates and cleans up its computer
after starting `apt` during the guest-network race.

- A fresh channel now verifies and retries the same container when LXC's first
daemonized start transiently loses its state-socket reply. The one-off
`wait_on_daemonized_start` error no longer makes the channel appear broken
while the automatic fleet pass successfully creates it moments later.

- 1Helm's narrow owned LXC forwarding rules are inserted ahead of host firewall
policies such as Docker's `FORWARD=DROP`, allowing resident guests to reach
package mirrors without replacing Docker or Tailscale chains.
Expand Down
27 changes: 21 additions & 6 deletions scripts/1helm-lxc-runtime
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,13 +78,28 @@ cpu_count() {
printf '%s' "$total"
}
start() {
[[ "$(state "$1")" == "RUNNING" ]] || lxc-start -P "$LXC_PATH" -n "$1" -d
local attempt
for attempt in {1..200}; do
[[ "$(state "$1")" == "RUNNING" ]] && return
sleep 0.1
local name="$1" start_attempt state_attempt detail=""
[[ "$(state "$name")" == "RUNNING" ]] && return
# A freshly unpacked container can hit LXC's daemonized-start socket race:
# lxc-start exits after wait_on_daemonized_start loses the state message even
# though the container definition is intact and immediately startable. Check
# the authoritative state after every invocation and retry that same exact
# validated machine instead of failing channel creation on the transient.
for start_attempt in {1..3}; do
if ! detail="$(lxc-start -P "$LXC_PATH" -n "$name" -d 2>&1)"; then
for state_attempt in {1..50}; do
[[ "$(state "$name")" == "RUNNING" ]] && return
sleep 0.1
done
continue
Comment on lines +90 to +94

@coderabbitai coderabbitai Bot Jul 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the relevant script and nearby logic.
git ls-files scripts/1helm-lxc-runtime
wc -l scripts/1helm-lxc-runtime
cat -n scripts/1helm-lxc-runtime | sed -n '1,170p'

Repository: gitcommit90/1Helm

Length of output: 8903

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Show only the relevant section if the file exists.
sed -n '1,170p' scripts/1helm-lxc-runtime | nl -ba | sed -n '70,120p'

Repository: gitcommit90/1Helm

Length of output: 195

🏁 Script executed:

#!/bin/bash
set -euo pipefail
sed -n '1,170p' scripts/1helm-lxc-runtime | nl -ba | sed -n '80,115p'

Repository: gitcommit90/1Helm

Length of output: 195

🏁 Script executed:

#!/bin/bash
set -euo pipefail
if [[ -f scripts/1helm-lxc-runtime ]]; then
  echo "FOUND"
  nl -ba scripts/1helm-lxc-runtime | sed -n '75,110p'
else
  echo "MISSING"
  find . -maxdepth 3 -type f | sed -n '1,50p'
fi

Repository: gitcommit90/1Helm

Length of output: 201

🌐 Web query:

lxc-start daemonized exit code socket response still starting RUNNING state polling behavior

💡 Result:

When a container is started with lxc-start in daemonized mode, LXC employs a specific synchronization mechanism to ensure the caller can reliably detect when the container has successfully reached the RUNNING state [1][2]. Mechanism Details: 1. Anonymous Socket Pair: To address race conditions where a container's init process might exit before the primary command socket is fully initialized, LXC creates a short-lived anonymous AF_UNIX socket pair (pipe-like behavior) during daemonized initialization [1][2]. 2. Synchronization: The handler uses this socket pair to pass state updates internally. The parent process (the caller) can monitor these sockets to detect the transition to RUNNING [1][2]. 3. State Polling: Tools and callers often interface with LXC's monitor system or command sockets to poll or subscribe to lifecycle state changes (STARTING, RUNNING, STOPPING, etc.) [3][4]. Historically, this relied on lxc-monitord, but modern LXC implementations use native command socket notifications where clients register to receive state change events directly from the container's handler [2][4][5]. 4. Exit Code Handling: If lxc-start fails during the startup phase, it returns specific exit codes to the caller [6]. Generally: - 0-126: Represents the exit code of the application running inside the container [6]. - 128+n: Indicates the application was terminated by signal n [6]. - 255: Indicates an internal LXC error (e.g., failed setup, monitor communication failure) [6]. If a container fails to reach the RUNNING state within a specified timeout (often 5 seconds), the startup process will report a failure to the user, preventing it from incorrectly reporting success for a container that crashed immediately upon launch [2][7]. High-level management tools, such as those used in Proxmox, often implement their own timeout-based polling logic against the monitor socket to ensure that hooks and post-start actions only execute after the container is confirmed to be running [3].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Find start() callers and any nearby comments about timing/races.
rg -n "start\(" scripts/1helm-lxc-runtime
rg -n "socket race|RUNNING|50}|200}" scripts/1helm-lxc-runtime

Repository: gitcommit90/1Helm

Length of output: 1048

🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path
path = Path('scripts/1helm-lxc-runtime')
for i, line in enumerate(path.read_text().splitlines(), 1):
    if 'start(' in line or 'RUNNING' in line or 'lxc-start' in line:
        print(f"{i}:{line}")
PY

Repository: gitcommit90/1Helm

Length of output: 1210

🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path
path = Path('scripts/1helm-lxc-runtime')
lines = path.read_text().splitlines()
for i, line in enumerate(lines, 1):
    if 'start(' in line or 'RUNNING' in line or 'lxc-start' in line:
        print(f"{i}:{line}")
PY

Repository: gitcommit90/1Helm

Length of output: 1210

scripts/1helm-lxc-runtime:90-94 — Keep the longer polling window before retrying. The 5s wait after a non-zero lxc-start exit is shorter than the 20s success-path poll, so a daemonized start that lost its state message can still be in flight when the next attempt fires and exhaust all 3 tries early. Use the longer window before retrying.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/1helm-lxc-runtime` around lines 90 - 94, The retry path after a
non-zero lxc-start exit in the state polling loop must use the same longer
polling window as the success path. Update the loop around state polling to wait
up to 20 seconds before executing continue, preserving the existing RUNNING
check and retry behavior.

fi
for state_attempt in {1..200}; do
[[ "$(state "$name")" == "RUNNING" ]] && return
sleep 0.1
done
done
die "container did not start"
[[ -z "$detail" ]] || printf '%s\n' "$detail" >&2
Comment on lines +81 to +101

@coderabbitai coderabbitai Bot Jul 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Preserve startup diagnostics across attempts.

detail is overwritten on every lxc-start invocation. A later empty-output attempt can erase a useful error from an earlier failed attempt, causing the final failure to omit the captured startup details. Store per-attempt output and retain the last non-empty message or append all attempt details.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/1helm-lxc-runtime` around lines 81 - 101, The start retry loop in the
script overwrites the captured lxc-start diagnostics on each invocation,
allowing a later empty result to erase useful failure details. Update the
variables used by the retry loop around lxc-start so each attempt’s output is
preserved, retaining the last non-empty message or accumulating all non-empty
attempt details before the final diagnostic print.

die "container did not start after 3 verified attempts"
}
wait_for_guest_network() {
local name="$1" attempt
Expand Down
1 change: 1 addition & 0 deletions test/site.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,7 @@ test("installer assets are explicit and syntax-valid", () => {
assert.match(lxcHelper, /wait_for_guest_network[\s\S]*10\\\.0\\\.3\\\.[\s\S]*via 10\\\.0\\\.3\\\.1 dev eth0/, "fresh LXC provisioning waits for its DHCP address and default route");
assert.match(lxcHelper, /archive\.ubuntu\.com[\s\S]*security\.ubuntu\.com[\s\S]*curl -4[\s\S]*InRelease/, "fresh LXC provisioning proves DNS and outbound IPv4 HTTP before package bootstrap");
assert.match(lxcHelper, /start "\$name"[\s\S]*wait_for_guest_network "\$name"[\s\S]*apt-get update -o Acquire::ForceIPv4=true -o APT::Update::Error-Mode=any/, "package bootstrap begins only after bounded guest-network readiness");
assert.match(lxcHelper, /for start_attempt in \{1\.\.3\}[\s\S]*lxc-start[\s\S]*state "\$name"[\s\S]*continue[\s\S]*container did not start after 3 verified attempts/, "a transient daemonized LXC state-socket failure is retried against the same validated machine");
assert.match(lxcHelper, /cpuset\.cpus\.effective/, "LXC CPU limits are selected from the service's actually delegated host CPUs");
assert.match(lxcHelper, /cpu_count/, "LXC inspection counts noncontiguous delegated CPU lists correctly");
assert.match(lxcNetwork, /1helm-lxc-net-owned/, "the bridge wrapper stops only a bridge it started");
Expand Down
Loading