From c9e88499f776e03e99405e103aa6b47ab0a463c9 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 27 Jul 2026 19:58:41 +0000 Subject: [PATCH 1/3] Initial plan From 960adf17b890b5e3dae4ece63e1c13aa33d76e70 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 27 Jul 2026 20:03:32 +0000 Subject: [PATCH 2/3] docs: Runner Doctor B13 update + B15/B16 new failure modes (2026-07-27 scan) --- .github/agents/self-hosted-runner-doctor.md | 20 +++++++++++++++---- .../workflows/self-hosted-runner-doctor.md | 10 ++++++++-- .../shared/self-hosted-failure-modes.md | 10 ++++++++-- 3 files changed, 32 insertions(+), 8 deletions(-) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index d49d98afa..2f164c58d 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -81,7 +81,9 @@ Prefer the narrowest match. Examples: - Copilot CLI exits immediately (exit code 1, ~0.5 s, zero stdout/stderr) after AWF upgrade on Docker or gVisor but not sbx → B14 (`~/.copilot/config.json` incorrectly added to credential deny list; fixed in github/gh-aw-firewall#6374) - `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) - `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) -- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438) +- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap) +- `--network-isolation is not yet supported with --enable-host-access` → B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; root cause unresolved in github/gh-aw-firewall#6651) +- AWF rejects `--mount` with "host path must be absolute" and the path visibly contains `${VAR_NAME}` unexpanded → B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; root cause unresolved in github/gh-aw-firewall#6649) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) - `FATAL: http_port: IPv6 is not available` → B3 @@ -115,10 +117,14 @@ B11 / github/gh-aw-firewall#6072 — Rootless permission-repair diagnostics were B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix. `detectDnsResolutionFailure()` now augments the startup error with the unresolved host and recommends using the proxy IP or `dockerd --dns `. -B13 / github/gh-aw-firewall#6189, github/gh-aw-firewall#6438 — In `--network-isolation` mode, MCP tool calls to topology-attached peers or to a `difcProxyHost` were failing because those hosts were not included in `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** topology peer hostnames are added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of `:port`) is also added to `NO_PROXY`. Upgrade to AWF including both PRs. +B13 / github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 — In `--network-isolation` mode, MCP tool calls to topology-attached peers or to a `difcProxyHost` were failing because those hosts were not included in `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** topology peer hostnames are added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of `:port`) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist. Upgrade to AWF including all three PRs. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation. B14 / github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 — After the centralized mount-policy refactor (github/gh-aw-firewall#6339), `~/.copilot/config.json` was incorrectly added to the credential deny list, causing AWF to overlay it with a read-only `/dev/null` bind mount. The Copilot CLI atomically rewrites this file at startup via temp-file + rename; the read-only overlay makes the rename fail and the CLI exits silently (exit 1, ~0.5 s). **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `config.json` is removed from the deny list and restored to read-write. Workaround (older AWF): downgrade past github/gh-aw-firewall#6339 or pass `--keep-containers` and inspect for `EROFS`/rename errors. +B15 / github/gh-aw-firewall#6651 — AWF startup fails with `--network-isolation is not yet supported with --enable-host-access` when `localhost` is in the domain allowlist. The gh-aw compiler (v0.82.x+) auto-enables `--enable-host-access` when `localhost` is in the allowlist and also emits `network.isolation: true` + `topologyAttach` unconditionally; the two flags are mutually exclusive in `src/cli.ts` with no overlap support. **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required. + +B16 / github/gh-aw-firewall#6649 — AWF rejects `--mount` entries whose host path contains an unexpanded `${VAR}` reference (for example `${TERRAFORM_CLI_PATH}/terraform:/usr/local/bin/terraform`) with a "host path must be absolute" error. The gh-aw compiler wraps `sandbox.agent.mounts` specs containing `${}` in single quotes in the generated shell invocation; Bash single quotes prevent variable substitution, so the literal string reaches AWF's volume validator instead of the resolved path. **Unresolved** (github/gh-aw-firewall#6649 open). Workaround: replace `${VAR}` references with their literal values in `sandbox.agent.mounts`, or stage the path into a wrapper script that expands before calling AWF. + D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. @@ -212,8 +218,10 @@ Establish these facts before matching a failure mode: | B10 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` × 3 directories, each taking exactly ~30 s; total ~90 s wasted; cascading `EACCES: permission denied` / `Failed to remove chroot home directory after permission repair` | `fixArtifactPermissionsForRootless()` builds compound `tag@digest` image refs (e.g. `agent:0.27.22@sha256:55f065...`) for its `docker run --pull never` repair container. Despite `--pull never`, Docker still attempts registry manifest verification for compound refs; when GHCR credentials are unavailable (cleaned in an earlier workflow step or expired) the verification TCP-connects and times out (~30 s per directory). | **Fixed in AWF (PR github/gh-aw-firewall#6025)**: `resolvePermFixerImageRef()` now strips the `@sha256:...` digest and uses a tag-only ref. `--pull never` with a tag-only ref skips all registry I/O. Upgrade to AWF version that includes github/gh-aw-firewall#6025. Workaround (older AWF): ensure GHCR credentials are available until after the AWF cleanup step. **Additional fix (PRs github/gh-aw-firewall#6342 / github/gh-aw-firewall#6356, merged 2026-07-18):** Even with a tag-only image ref, `docker run` was missing `--entrypoint sh`, causing AWF's `entrypoint.sh` to execute in place of the repair command. `entrypoint.sh` waits ~30 s for an iptables-init container that never starts in this context, then fails. The fix adds `--entrypoint sh` and passes the `chown/chmod` command via `-c`; stdout is now captured alongside stderr so timeout messages surfaced on stdout are not silently swallowed. | `[WARN] Rootless artifact permission repair failed` messages each followed by exactly ~30 s delay in the workflow log; `docker pull --dry-run agent:...@sha256:...` times out while `docker inspect agent:...` succeeds | github/gh-aw-firewall#6025, github/gh-aw-firewall#6342, github/gh-aw-firewall#6356 | | B11 | AWF exits with code 1 and logs show `[WARN] Rootless artifact permission repair failed ... (exit 1)` plus cleanup warnings, but no actionable stderr detail from the repair container | `fixArtifactPermissionsForRootless()` previously discarded repair-container stderr, so the warning hid the real failure context. The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it. | **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured in the warning, and chroot-home cleanup noise is reduced by downgrading that log from `warn` to `debug`. Treat this mode as diagnostic-opacity around a pre-existing non-zero command exit, not cleanup-driven exit-code mutation. | Compare agent-command logs with `Command completed with exit code: `; on affected versions, repair warnings lack stderr context. On AWF including github/gh-aw-firewall#6072, the warning includes stderr detail for root-cause triage. | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 | | B12 | `getaddrinfo EAI_AGAIN ` or `ENOTFOUND ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` on ARC/DinD, and DinD containers cannot resolve the Kubernetes Service name even after the ordering fix | ARC/DinD DNS isolation: the DinD Docker network does not forward DNS to the cluster resolver, so Kubernetes Service names such as `awmg-cli-proxy` are unreachable from AWF containers even though the service exists. | **Diagnosed in AWF (PR github/gh-aw-firewall#6328, merged 2026-07-17):** `detectDnsResolutionFailure()` scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with concrete remediation. Address the DIFC proxy by IP, or configure `dockerd --dns ` so DinD containers can reach the cluster DNS server. | `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers; compare with the augmented startup error from `detectDnsResolutionFailure()` | github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | -| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473 | +| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the Firewall Issue Dispatcher block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 | | B14 | After upgrading to AWF including the centralized mount-policy refactor (PR github/gh-aw-firewall#6339), the Copilot CLI agent exits immediately (exit code 1, ~0.5 s, zero stdout/stderr); no network calls are made; the issue appears on Docker and gVisor runtimes but not sbx (which moves files aside rather than overlay-masking them) | PR github/gh-aw-firewall#6339 added `~/.copilot/config.json` to the credential **deny list** with reason *"Copilot CLI persisted auth token"*. This is incorrect: `config.json` contains only experiment flags, first-launch timestamps, and plugin metadata — no token. AWF masks denied files with a **read-only** `/dev/null` bind-mount overlay. The Copilot CLI reads **and atomically rewrites** `config.json` via temp-file + rename at startup; a rename cannot target a bind-mounted path and the overlay is read-only, so the CLI crashes silently. | **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `~/.copilot/config.json` is removed from the credential deny list; the file is bind-mounted read-write as part of the `.copilot` tool subdir (pre-#6339 behaviour). Upgrade to the AWF version including github/gh-aw-firewall#6374. Workaround (older AWF): pass `--keep-containers` and inspect the agent container for `EROFS` or rename errors; downgrade AWF past github/gh-aw-firewall#6339 if unpatched. | `awf ... copilot agent --version` — if it exits 1 with no output in ~0.5 s, confirm AWF version includes github/gh-aw-firewall#6374; inspect agent container logs for `EROFS`/`rename` errors on `~/.copilot/config.json` | github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 | +| B15 | AWF startup fails with `--network-isolation is not yet supported with --enable-host-access`; happens when `localhost` is in the domain allowlist because the gh-aw compiler auto-enables `--enable-host-access` and also emits `network.isolation: true` + `topologyAttach` | In `src/cli.ts`, `--network-isolation` and `--enable-host-access` are mutually exclusive with no overlap support. The gh-aw compiler (v0.82.x+) does not detect the conflict: it emits `isolation:true` + `topologyAttach` unconditionally while also auto-enabling `--enable-host-access` when `localhost` is in the allowlist. The compiled workflow always fails at runtime. | **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required. | `awf --network-isolation --enable-host-access ...` — reproduces immediately with `not yet supported` error; check compiler output for both flags being emitted together when `localhost` is in allowlist | github/gh-aw-firewall#6651 | +| B16 | AWF rejects a `--mount` entry with a non-absolute host path error; the mount spec was defined as `${TERRAFORM_CLI_PATH}/terraform:...` (or similar env-var pattern) but the literal string `${TERRAFORM_CLI_PATH}` reaches AWF's volume validator | The gh-aw compiler wraps `sandbox.agent.mounts` specs containing `${}` references in single quotes in the generated shell invocation. Bash single quotes prevent variable substitution, so the literal variable reference string (e.g. `${TERRAFORM_CLI_PATH}`) reaches AWF's volume validation instead of the resolved absolute path. | **Unresolved** (github/gh-aw-firewall#6649 open). Workaround: replace `${VAR}` references with their literal values in `sandbox.agent.mounts`, or stage the path into a wrapper script that expands before calling AWF. | Inspect the generated `awf` invocation in the compiled lock file — if mount specs appear in single quotes containing `${...}`, the bug is present. Running `awf ... --mount '${VAR}/path:/dest'` reproduces the rejection. | github/gh-aw-firewall#6649 | ## Category C — GHES / GHEC / `ghe.com` @@ -282,7 +290,9 @@ Establish these facts before matching a failure mode: | Copilot CLI exits immediately (exit 1, ~0.5 s, zero stdout/stderr) after AWF upgrade; works in sbx but not Docker/gVisor | B14 (`.copilot/config.json` masked as credential; fixed in github/gh-aw-firewall#6374) | | `copilot: command not found` inside sbx microVM when binary is at `~/.local/bin/copilot` | D10 (`bash -lc` login init resets injected PATH; fixed by command wrapper `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) | | `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) | -| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438) | +| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap) | +| `--network-isolation is not yet supported with --enable-host-access` | B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; root cause unresolved in github/gh-aw-firewall#6651) | +| AWF rejects `--mount` with "host path must be absolute" (or similar) and the path visibly contains `${VAR_NAME}` unexpanded | B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; root cause unresolved in github/gh-aw-firewall#6649) | ## Known unresolved items @@ -294,3 +304,5 @@ Flag these explicitly instead of implying there is a complete fix: - C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes - C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911) - D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause (SIGABRT `StringBytes::Encode`); one-shot retry mitigates (~8% failure rate) but does not prevent the crash +- B15 / github/gh-aw-firewall#6651 — `--network-isolation` + `--enable-host-access` mutual-exclusion conflict when `localhost` in domain allowlist triggers auto-enable; fix requires either compiler guard or AWF relaxation +- B16 / github/gh-aw-firewall#6649 — `${VAR}` in `sandbox.agent.mounts` passed literally by single-quote wrapping in compiler; AWF rejects non-absolute host paths diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 08bf79026..432886cdb 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -110,7 +110,9 @@ Prefer the narrowest match. Examples: - Copilot CLI exits immediately (exit code 1, ~0.5 s, zero stdout/stderr) after AWF upgrade on Docker or gVisor but not sbx → B14 (`~/.copilot/config.json` incorrectly added to credential deny list; fixed in github/gh-aw-firewall#6374) - `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) - `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) -- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438) +- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap) +- `--network-isolation is not yet supported with --enable-host-access` → B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; root cause unresolved in github/gh-aw-firewall#6651) +- AWF rejects `--mount` with "host path must be absolute" and the path visibly contains `${VAR_NAME}` unexpanded → B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; root cause unresolved in github/gh-aw-firewall#6649) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) - `FATAL: http_port: IPv6 is not available` → B3 @@ -144,10 +146,14 @@ B11 / github/gh-aw-firewall#6072 — Rootless permission-repair diagnostics were B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix. `detectDnsResolutionFailure()` now augments the startup error with the unresolved host and recommends using the proxy IP or `dockerd --dns `. -B13 / github/gh-aw-firewall#6189, github/gh-aw-firewall#6438 — In `--network-isolation` mode, MCP tool calls to topology-attached peers or to a `difcProxyHost` were failing because those hosts were not included in `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** topology peer hostnames are added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of `:port`) is also added to `NO_PROXY`. Upgrade to AWF including both PRs. +B13 / github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 — In `--network-isolation` mode, MCP tool calls to topology-attached peers or to a `difcProxyHost` were failing because those hosts were not included in `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** topology peer hostnames are added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of `:port`) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist. Upgrade to AWF including all three PRs. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation. B14 / github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 — After the centralized mount-policy refactor (github/gh-aw-firewall#6339), `~/.copilot/config.json` was incorrectly added to the credential deny list, causing AWF to overlay it with a read-only `/dev/null` bind mount. The Copilot CLI atomically rewrites this file at startup via temp-file + rename; the read-only overlay makes the rename fail and the CLI exits silently (exit 1, ~0.5 s). **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `config.json` is removed from the deny list and restored to read-write. Workaround (older AWF): downgrade past github/gh-aw-firewall#6339 or pass `--keep-containers` and inspect for `EROFS`/rename errors. +B15 / github/gh-aw-firewall#6651 — AWF startup fails with `--network-isolation is not yet supported with --enable-host-access` when `localhost` is in the domain allowlist. The gh-aw compiler (v0.82.x+) auto-enables `--enable-host-access` when `localhost` is in the allowlist and also emits `network.isolation: true` + `topologyAttach` unconditionally; the two flags are mutually exclusive in `src/cli.ts` with no overlap support. **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required. + +B16 / github/gh-aw-firewall#6649 — AWF rejects `--mount` entries whose host path contains an unexpanded `${VAR}` reference (for example `${TERRAFORM_CLI_PATH}/terraform:/usr/local/bin/terraform`) with a "host path must be absolute" error. The gh-aw compiler wraps `sandbox.agent.mounts` specs containing `${}` in single quotes in the generated shell invocation; Bash single quotes prevent variable substitution, so the literal string reaches AWF's volume validator instead of the resolved path. **Unresolved** (github/gh-aw-firewall#6649 open). Workaround: replace `${VAR}` references with their literal values in `sandbox.agent.mounts`, or stage the path into a wrapper script that expands before calling AWF. + D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index f1d08a2d6..ab20a3edd 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -51,8 +51,10 @@ Establish these facts before matching a failure mode: | B10 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` × 3 directories, each taking exactly ~30 s; total ~90 s wasted; cascading `EACCES: permission denied` / `Failed to remove chroot home directory after permission repair` | `fixArtifactPermissionsForRootless()` builds compound `tag@digest` image refs (e.g. `agent:0.27.22@sha256:55f065...`) for its `docker run --pull never` repair container. Despite `--pull never`, Docker still attempts registry manifest verification for compound refs; when GHCR credentials are unavailable (cleaned in an earlier workflow step or expired) the verification TCP-connects and times out (~30 s per directory). | **Fixed in AWF (PR github/gh-aw-firewall#6025)**: `resolvePermFixerImageRef()` now strips the `@sha256:...` digest and uses a tag-only ref. `--pull never` with a tag-only ref skips all registry I/O. Upgrade to AWF version that includes github/gh-aw-firewall#6025. Workaround (older AWF): ensure GHCR credentials are available until after the AWF cleanup step. **Additional fix (PRs github/gh-aw-firewall#6342 / github/gh-aw-firewall#6356, merged 2026-07-18):** Even with a tag-only image ref, `docker run` was missing `--entrypoint sh`, causing AWF's `entrypoint.sh` to execute in place of the repair command. `entrypoint.sh` waits ~30 s for an iptables-init container that never starts in this context, then fails. The fix adds `--entrypoint sh` and passes the `chown/chmod` command via `-c`; stdout is now captured alongside stderr so timeout messages surfaced on stdout are not silently swallowed. | `[WARN] Rootless artifact permission repair failed` messages each followed by exactly ~30 s delay in the workflow log; `docker pull --dry-run agent:...@sha256:...` times out while `docker inspect agent:...` succeeds | github/gh-aw-firewall#6025, github/gh-aw-firewall#6342, github/gh-aw-firewall#6356 | | B11 | AWF exits with code 1 and logs show `[WARN] Rootless artifact permission repair failed ... (exit 1)` plus cleanup warnings, but no actionable stderr detail from the repair container | `fixArtifactPermissionsForRootless()` previously discarded repair-container stderr, so the warning hid the real failure context. The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it. | **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured in the warning, and chroot-home cleanup noise is reduced by downgrading that log from `warn` to `debug`. Treat this mode as diagnostic-opacity around a pre-existing non-zero command exit, not cleanup-driven exit-code mutation. | Compare agent-command logs with `Command completed with exit code: `; on affected versions, repair warnings lack stderr context. On AWF including github/gh-aw-firewall#6072, the warning includes stderr detail for root-cause triage. | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 | | B12 | `getaddrinfo EAI_AGAIN ` or `ENOTFOUND ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` on ARC/DinD, and DinD containers cannot resolve the Kubernetes Service name even after the ordering fix | ARC/DinD DNS isolation: the DinD Docker network does not forward DNS to the cluster resolver, so Kubernetes Service names such as `awmg-cli-proxy` are unreachable from AWF containers even though the service exists. | **Diagnosed in AWF (PR github/gh-aw-firewall#6328, merged 2026-07-17):** `detectDnsResolutionFailure()` scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with concrete remediation. Address the DIFC proxy by IP, or configure `dockerd --dns ` so DinD containers can reach the cluster DNS server. | `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers; compare with the augmented startup error from `detectDnsResolutionFailure()` | github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | -| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473 | +| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the Firewall Issue Dispatcher block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 | | B14 | After upgrading to AWF including the centralized mount-policy refactor (PR github/gh-aw-firewall#6339), the Copilot CLI agent exits immediately (exit code 1, ~0.5 s, zero stdout/stderr); no network calls are made; the issue appears on Docker and gVisor runtimes but not sbx (which moves files aside rather than overlay-masking them) | PR github/gh-aw-firewall#6339 added `~/.copilot/config.json` to the credential **deny list** with reason *"Copilot CLI persisted auth token"*. This is incorrect: `config.json` contains only experiment flags, first-launch timestamps, and plugin metadata — no token. AWF masks denied files with a **read-only** `/dev/null` bind-mount overlay. The Copilot CLI reads **and atomically rewrites** `config.json` via temp-file + rename at startup; a rename cannot target a bind-mounted path and the overlay is read-only, so the CLI crashes silently. | **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `~/.copilot/config.json` is removed from the credential deny list; the file is bind-mounted read-write as part of the `.copilot` tool subdir (pre-#6339 behaviour). Upgrade to the AWF version including github/gh-aw-firewall#6374. Workaround (older AWF): pass `--keep-containers` and inspect the agent container for `EROFS` or rename errors; downgrade AWF past github/gh-aw-firewall#6339 if unpatched. | `awf ... copilot agent --version` — if it exits 1 with no output in ~0.5 s, confirm AWF version includes github/gh-aw-firewall#6374; inspect agent container logs for `EROFS`/`rename` errors on `~/.copilot/config.json` | github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 | +| B15 | AWF startup fails with `--network-isolation is not yet supported with --enable-host-access`; happens when `localhost` is in the domain allowlist because the gh-aw compiler auto-enables `--enable-host-access` and also emits `network.isolation: true` + `topologyAttach` | In `src/cli.ts`, `--network-isolation` and `--enable-host-access` are mutually exclusive with no overlap support. The gh-aw compiler (v0.82.x+) does not detect the conflict: it emits `isolation:true` + `topologyAttach` unconditionally while also auto-enabling `--enable-host-access` when `localhost` is in the allowlist. The compiled workflow always fails at runtime. | **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required. | `awf --network-isolation --enable-host-access ...` — reproduces immediately with `not yet supported` error; check compiler output for both flags being emitted together when `localhost` is in allowlist | github/gh-aw-firewall#6651 | +| B16 | AWF rejects a `--mount` entry with a non-absolute host path error; the mount spec was defined as `${TERRAFORM_CLI_PATH}/terraform:...` (or similar env-var pattern) but the literal string `${TERRAFORM_CLI_PATH}` reaches AWF's volume validator | The gh-aw compiler wraps `sandbox.agent.mounts` specs containing `${}` references in single quotes in the generated shell invocation. Bash single quotes prevent variable substitution, so the literal variable reference string (e.g. `${TERRAFORM_CLI_PATH}`) reaches AWF's volume validation instead of the resolved absolute path. | **Unresolved** (github/gh-aw-firewall#6649 open). Workaround: replace `${VAR}` references with their literal values in `sandbox.agent.mounts`, or stage the path into a wrapper script that expands before calling AWF. | Inspect the generated `awf` invocation in the compiled lock file — if mount specs appear in single quotes containing `${...}`, the bug is present. Running `awf ... --mount '${VAR}/path:/dest'` reproduces the rejection. | github/gh-aw-firewall#6649 | ## Category C — GHES / GHEC / `ghe.com` @@ -121,7 +123,9 @@ Establish these facts before matching a failure mode: | Copilot CLI exits immediately (exit 1, ~0.5 s, zero stdout/stderr) after AWF upgrade; works in sbx but not Docker/gVisor | B14 (`.copilot/config.json` masked as credential; fixed in github/gh-aw-firewall#6374) | | `copilot: command not found` inside sbx microVM when binary is at `~/.local/bin/copilot` | D10 (`bash -lc` login init resets injected PATH; fixed by command wrapper `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) | | `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) | -| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438) | +| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap) | +| `--network-isolation is not yet supported with --enable-host-access` | B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; root cause unresolved in github/gh-aw-firewall#6651) | +| AWF rejects `--mount` with "host path must be absolute" (or similar) and the path visibly contains `${VAR_NAME}` unexpanded | B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; root cause unresolved in github/gh-aw-firewall#6649) | ## Known unresolved items @@ -133,3 +137,5 @@ Flag these explicitly instead of implying there is a complete fix: - C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes - C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911) - D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause (SIGABRT `StringBytes::Encode`); one-shot retry mitigates (~8% failure rate) but does not prevent the crash +- B15 / github/gh-aw-firewall#6651 — `--network-isolation` + `--enable-host-access` mutual-exclusion conflict when `localhost` in domain allowlist triggers auto-enable; fix requires either compiler guard or AWF relaxation +- B16 / github/gh-aw-firewall#6649 — `${VAR}` in `sandbox.agent.mounts` passed literally by single-quote wrapping in compiler; AWF rejects non-absolute host paths From 8a462fb2174dbcb59a45b104486299c6bf91e338 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 27 Jul 2026 23:16:56 +0000 Subject: [PATCH 3/3] docs: fix B13 diagnosis and B15 workaround per review feedback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - B13: reword 'remaining Squid ACL gap' as audit/policy-manifest reporting false positive in all 3 files (category table, quick-lookup, and condensed-description sections); add #6658 as the tracking PR - B15: add security.legacySecurity: true to the network.isolation: false workaround in all 3 files — applySecurityMode() forces isolation back on and clears enableHostAccess in default strict mode without it --- .github/agents/self-hosted-runner-doctor.md | 12 ++++++------ .github/workflows/self-hosted-runner-doctor.md | 6 +++--- .../workflows/shared/self-hosted-failure-modes.md | 6 +++--- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index 2f164c58d..a1aeee87f 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -81,7 +81,7 @@ Prefer the narrowest match. Examples: - Copilot CLI exits immediately (exit code 1, ~0.5 s, zero stdout/stderr) after AWF upgrade on Docker or gVisor but not sbx → B14 (`~/.copilot/config.json` incorrectly added to credential deny list; fixed in github/gh-aw-firewall#6374) - `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) - `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) -- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap) +- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) - `--network-isolation is not yet supported with --enable-host-access` → B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; root cause unresolved in github/gh-aw-firewall#6651) - AWF rejects `--mount` with "host path must be absolute" and the path visibly contains `${VAR_NAME}` unexpanded → B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; root cause unresolved in github/gh-aw-firewall#6649) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) @@ -117,11 +117,11 @@ B11 / github/gh-aw-firewall#6072 — Rootless permission-repair diagnostics were B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix. `detectDnsResolutionFailure()` now augments the startup error with the unresolved host and recommends using the proxy IP or `dockerd --dns `. -B13 / github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 — In `--network-isolation` mode, MCP tool calls to topology-attached peers or to a `difcProxyHost` were failing because those hosts were not included in `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** topology peer hostnames are added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of `:port`) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist. Upgrade to AWF including all three PRs. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation. +B13 / github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652, github/gh-aw-firewall#6658 — In `--network-isolation` mode, MCP tool calls to topology-attached peers or to a `difcProxyHost` were failing because those hosts were not included in `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** topology peer hostnames are added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of `:port`) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist. Upgrade to AWF including all three PRs. **Open sub-issue (github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658):** After github/gh-aw-firewall#6473, topology peer hostnames receive runtime Squid allow rules and are not actually blocked; however, the Firewall Issue Dispatcher audit/block report is generated from the policy manifest (not the live Squid config) and does not include topology-peer entries — causing `awmg-mcpg` to appear as denied when traffic is permitted (**reporting false positive**). The report also incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Tracked in github/gh-aw-firewall#6658. B14 / github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 — After the centralized mount-policy refactor (github/gh-aw-firewall#6339), `~/.copilot/config.json` was incorrectly added to the credential deny list, causing AWF to overlay it with a read-only `/dev/null` bind mount. The Copilot CLI atomically rewrites this file at startup via temp-file + rename; the read-only overlay makes the rename fail and the CLI exits silently (exit 1, ~0.5 s). **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `config.json` is removed from the deny list and restored to read-write. Workaround (older AWF): downgrade past github/gh-aw-firewall#6339 or pass `--keep-containers` and inspect for `EROFS`/rename errors. -B15 / github/gh-aw-firewall#6651 — AWF startup fails with `--network-isolation is not yet supported with --enable-host-access` when `localhost` is in the domain allowlist. The gh-aw compiler (v0.82.x+) auto-enables `--enable-host-access` when `localhost` is in the allowlist and also emits `network.isolation: true` + `topologyAttach` unconditionally; the two flags are mutually exclusive in `src/cli.ts` with no overlap support. **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required. +B15 / github/gh-aw-firewall#6651 — AWF startup fails with `--network-isolation is not yet supported with --enable-host-access` when `localhost` is in the domain allowlist. The gh-aw compiler (v0.82.x+) auto-enables `--enable-host-access` when `localhost` is in the allowlist and also emits `network.isolation: true` + `topologyAttach` unconditionally; the two flags are mutually exclusive in `src/cli.ts` with no overlap support. **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set both `network.isolation: false` **and** `security.legacySecurity: true` in the AWF config if host-access is required — `network.isolation: false` alone is insufficient because `applySecurityMode()` forces isolation back on and clears `enableHostAccess` in default strict mode (`src/commands/validators/security-mode.ts:37-74`). B16 / github/gh-aw-firewall#6649 — AWF rejects `--mount` entries whose host path contains an unexpanded `${VAR}` reference (for example `${TERRAFORM_CLI_PATH}/terraform:/usr/local/bin/terraform`) with a "host path must be absolute" error. The gh-aw compiler wraps `sandbox.agent.mounts` specs containing `${}` in single quotes in the generated shell invocation; Bash single quotes prevent variable substitution, so the literal string reaches AWF's volume validator instead of the resolved path. **Unresolved** (github/gh-aw-firewall#6649 open). Workaround: replace `${VAR}` references with their literal values in `sandbox.agent.mounts`, or stage the path into a wrapper script that expands before calling AWF. @@ -218,9 +218,9 @@ Establish these facts before matching a failure mode: | B10 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` × 3 directories, each taking exactly ~30 s; total ~90 s wasted; cascading `EACCES: permission denied` / `Failed to remove chroot home directory after permission repair` | `fixArtifactPermissionsForRootless()` builds compound `tag@digest` image refs (e.g. `agent:0.27.22@sha256:55f065...`) for its `docker run --pull never` repair container. Despite `--pull never`, Docker still attempts registry manifest verification for compound refs; when GHCR credentials are unavailable (cleaned in an earlier workflow step or expired) the verification TCP-connects and times out (~30 s per directory). | **Fixed in AWF (PR github/gh-aw-firewall#6025)**: `resolvePermFixerImageRef()` now strips the `@sha256:...` digest and uses a tag-only ref. `--pull never` with a tag-only ref skips all registry I/O. Upgrade to AWF version that includes github/gh-aw-firewall#6025. Workaround (older AWF): ensure GHCR credentials are available until after the AWF cleanup step. **Additional fix (PRs github/gh-aw-firewall#6342 / github/gh-aw-firewall#6356, merged 2026-07-18):** Even with a tag-only image ref, `docker run` was missing `--entrypoint sh`, causing AWF's `entrypoint.sh` to execute in place of the repair command. `entrypoint.sh` waits ~30 s for an iptables-init container that never starts in this context, then fails. The fix adds `--entrypoint sh` and passes the `chown/chmod` command via `-c`; stdout is now captured alongside stderr so timeout messages surfaced on stdout are not silently swallowed. | `[WARN] Rootless artifact permission repair failed` messages each followed by exactly ~30 s delay in the workflow log; `docker pull --dry-run agent:...@sha256:...` times out while `docker inspect agent:...` succeeds | github/gh-aw-firewall#6025, github/gh-aw-firewall#6342, github/gh-aw-firewall#6356 | | B11 | AWF exits with code 1 and logs show `[WARN] Rootless artifact permission repair failed ... (exit 1)` plus cleanup warnings, but no actionable stderr detail from the repair container | `fixArtifactPermissionsForRootless()` previously discarded repair-container stderr, so the warning hid the real failure context. The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it. | **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured in the warning, and chroot-home cleanup noise is reduced by downgrading that log from `warn` to `debug`. Treat this mode as diagnostic-opacity around a pre-existing non-zero command exit, not cleanup-driven exit-code mutation. | Compare agent-command logs with `Command completed with exit code: `; on affected versions, repair warnings lack stderr context. On AWF including github/gh-aw-firewall#6072, the warning includes stderr detail for root-cause triage. | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 | | B12 | `getaddrinfo EAI_AGAIN ` or `ENOTFOUND ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` on ARC/DinD, and DinD containers cannot resolve the Kubernetes Service name even after the ordering fix | ARC/DinD DNS isolation: the DinD Docker network does not forward DNS to the cluster resolver, so Kubernetes Service names such as `awmg-cli-proxy` are unreachable from AWF containers even though the service exists. | **Diagnosed in AWF (PR github/gh-aw-firewall#6328, merged 2026-07-17):** `detectDnsResolutionFailure()` scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with concrete remediation. Address the DIFC proxy by IP, or configure `dockerd --dns ` so DinD containers can reach the cluster DNS server. | `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers; compare with the augmented startup error from `detectDnsResolutionFailure()` | github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | -| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the Firewall Issue Dispatcher block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 | +| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. **Open sub-issue (github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658):** After github/gh-aw-firewall#6473, topology peer hostnames receive runtime Squid allow rules and traffic is permitted at the network level. However, the Firewall Issue Dispatcher **audit/block report** is generated from the policy manifest rather than the live Squid config; the manifest does not yet include topology-peer entries, so the block report incorrectly flags `awmg-mcpg` as denied. This is a **reporting false positive** — actual traffic is not blocked. The report also incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Tracked in github/gh-aw-firewall#6658. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .` after github/gh-aw-firewall#6473; if the Firewall Issue Dispatcher block report flags a topology peer after github/gh-aw-firewall#6473, treat it as a reporting false positive (manifest attribution gap; tracked in github/gh-aw-firewall#6658) | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652, github/gh-aw-firewall#6658 | | B14 | After upgrading to AWF including the centralized mount-policy refactor (PR github/gh-aw-firewall#6339), the Copilot CLI agent exits immediately (exit code 1, ~0.5 s, zero stdout/stderr); no network calls are made; the issue appears on Docker and gVisor runtimes but not sbx (which moves files aside rather than overlay-masking them) | PR github/gh-aw-firewall#6339 added `~/.copilot/config.json` to the credential **deny list** with reason *"Copilot CLI persisted auth token"*. This is incorrect: `config.json` contains only experiment flags, first-launch timestamps, and plugin metadata — no token. AWF masks denied files with a **read-only** `/dev/null` bind-mount overlay. The Copilot CLI reads **and atomically rewrites** `config.json` via temp-file + rename at startup; a rename cannot target a bind-mounted path and the overlay is read-only, so the CLI crashes silently. | **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `~/.copilot/config.json` is removed from the credential deny list; the file is bind-mounted read-write as part of the `.copilot` tool subdir (pre-#6339 behaviour). Upgrade to the AWF version including github/gh-aw-firewall#6374. Workaround (older AWF): pass `--keep-containers` and inspect the agent container for `EROFS` or rename errors; downgrade AWF past github/gh-aw-firewall#6339 if unpatched. | `awf ... copilot agent --version` — if it exits 1 with no output in ~0.5 s, confirm AWF version includes github/gh-aw-firewall#6374; inspect agent container logs for `EROFS`/`rename` errors on `~/.copilot/config.json` | github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 | -| B15 | AWF startup fails with `--network-isolation is not yet supported with --enable-host-access`; happens when `localhost` is in the domain allowlist because the gh-aw compiler auto-enables `--enable-host-access` and also emits `network.isolation: true` + `topologyAttach` | In `src/cli.ts`, `--network-isolation` and `--enable-host-access` are mutually exclusive with no overlap support. The gh-aw compiler (v0.82.x+) does not detect the conflict: it emits `isolation:true` + `topologyAttach` unconditionally while also auto-enabling `--enable-host-access` when `localhost` is in the allowlist. The compiled workflow always fails at runtime. | **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required. | `awf --network-isolation --enable-host-access ...` — reproduces immediately with `not yet supported` error; check compiler output for both flags being emitted together when `localhost` is in allowlist | github/gh-aw-firewall#6651 | +| B15 | AWF startup fails with `--network-isolation is not yet supported with --enable-host-access`; happens when `localhost` is in the domain allowlist because the gh-aw compiler auto-enables `--enable-host-access` and also emits `network.isolation: true` + `topologyAttach` | In `src/cli.ts`, `--network-isolation` and `--enable-host-access` are mutually exclusive with no overlap support. The gh-aw compiler (v0.82.x+) does not detect the conflict: it emits `isolation:true` + `topologyAttach` unconditionally while also auto-enabling `--enable-host-access` when `localhost` is in the allowlist. The compiled workflow always fails at runtime. | **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set both `network.isolation: false` **and** `security.legacySecurity: true` in the AWF config if host-access is required — `network.isolation: false` alone is insufficient because `applySecurityMode()` forces isolation back on and clears `enableHostAccess` in default strict mode (`src/commands/validators/security-mode.ts:37-74`). | `awf --network-isolation --enable-host-access ...` — reproduces immediately with `not yet supported` error; check compiler output for both flags being emitted together when `localhost` is in allowlist | github/gh-aw-firewall#6651 | | B16 | AWF rejects a `--mount` entry with a non-absolute host path error; the mount spec was defined as `${TERRAFORM_CLI_PATH}/terraform:...` (or similar env-var pattern) but the literal string `${TERRAFORM_CLI_PATH}` reaches AWF's volume validator | The gh-aw compiler wraps `sandbox.agent.mounts` specs containing `${}` references in single quotes in the generated shell invocation. Bash single quotes prevent variable substitution, so the literal variable reference string (e.g. `${TERRAFORM_CLI_PATH}`) reaches AWF's volume validation instead of the resolved absolute path. | **Unresolved** (github/gh-aw-firewall#6649 open). Workaround: replace `${VAR}` references with their literal values in `sandbox.agent.mounts`, or stage the path into a wrapper script that expands before calling AWF. | Inspect the generated `awf` invocation in the compiled lock file — if mount specs appear in single quotes containing `${...}`, the bug is present. Running `awf ... --mount '${VAR}/path:/dest'` reproduces the rejection. | github/gh-aw-firewall#6649 | ## Category C — GHES / GHEC / `ghe.com` @@ -290,7 +290,7 @@ Establish these facts before matching a failure mode: | Copilot CLI exits immediately (exit 1, ~0.5 s, zero stdout/stderr) after AWF upgrade; works in sbx but not Docker/gVisor | B14 (`.copilot/config.json` masked as credential; fixed in github/gh-aw-firewall#6374) | | `copilot: command not found` inside sbx microVM when binary is at `~/.local/bin/copilot` | D10 (`bash -lc` login init resets injected PATH; fixed by command wrapper `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) | | `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) | -| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap) | +| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) | | `--network-isolation is not yet supported with --enable-host-access` | B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; root cause unresolved in github/gh-aw-firewall#6651) | | AWF rejects `--mount` with "host path must be absolute" (or similar) and the path visibly contains `${VAR_NAME}` unexpanded | B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; root cause unresolved in github/gh-aw-firewall#6649) | diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 432886cdb..6f44dc107 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -110,7 +110,7 @@ Prefer the narrowest match. Examples: - Copilot CLI exits immediately (exit code 1, ~0.5 s, zero stdout/stderr) after AWF upgrade on Docker or gVisor but not sbx → B14 (`~/.copilot/config.json` incorrectly added to credential deny list; fixed in github/gh-aw-firewall#6374) - `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) - `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) -- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap) +- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) - `--network-isolation is not yet supported with --enable-host-access` → B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; root cause unresolved in github/gh-aw-firewall#6651) - AWF rejects `--mount` with "host path must be absolute" and the path visibly contains `${VAR_NAME}` unexpanded → B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; root cause unresolved in github/gh-aw-firewall#6649) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) @@ -146,11 +146,11 @@ B11 / github/gh-aw-firewall#6072 — Rootless permission-repair diagnostics were B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix. `detectDnsResolutionFailure()` now augments the startup error with the unresolved host and recommends using the proxy IP or `dockerd --dns `. -B13 / github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 — In `--network-isolation` mode, MCP tool calls to topology-attached peers or to a `difcProxyHost` were failing because those hosts were not included in `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** topology peer hostnames are added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of `:port`) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist. Upgrade to AWF including all three PRs. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation. +B13 / github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652, github/gh-aw-firewall#6658 — In `--network-isolation` mode, MCP tool calls to topology-attached peers or to a `difcProxyHost` were failing because those hosts were not included in `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** topology peer hostnames are added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of `:port`) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist. Upgrade to AWF including all three PRs. **Open sub-issue (github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658):** After github/gh-aw-firewall#6473, topology peer hostnames receive runtime Squid allow rules and are not actually blocked; however, the Firewall Issue Dispatcher audit/block report is generated from the policy manifest (not the live Squid config) and does not include topology-peer entries — causing `awmg-mcpg` to appear as denied when traffic is permitted (**reporting false positive**). The report also incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Tracked in github/gh-aw-firewall#6658. B14 / github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 — After the centralized mount-policy refactor (github/gh-aw-firewall#6339), `~/.copilot/config.json` was incorrectly added to the credential deny list, causing AWF to overlay it with a read-only `/dev/null` bind mount. The Copilot CLI atomically rewrites this file at startup via temp-file + rename; the read-only overlay makes the rename fail and the CLI exits silently (exit 1, ~0.5 s). **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `config.json` is removed from the deny list and restored to read-write. Workaround (older AWF): downgrade past github/gh-aw-firewall#6339 or pass `--keep-containers` and inspect for `EROFS`/rename errors. -B15 / github/gh-aw-firewall#6651 — AWF startup fails with `--network-isolation is not yet supported with --enable-host-access` when `localhost` is in the domain allowlist. The gh-aw compiler (v0.82.x+) auto-enables `--enable-host-access` when `localhost` is in the allowlist and also emits `network.isolation: true` + `topologyAttach` unconditionally; the two flags are mutually exclusive in `src/cli.ts` with no overlap support. **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required. +B15 / github/gh-aw-firewall#6651 — AWF startup fails with `--network-isolation is not yet supported with --enable-host-access` when `localhost` is in the domain allowlist. The gh-aw compiler (v0.82.x+) auto-enables `--enable-host-access` when `localhost` is in the allowlist and also emits `network.isolation: true` + `topologyAttach` unconditionally; the two flags are mutually exclusive in `src/cli.ts` with no overlap support. **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set both `network.isolation: false` **and** `security.legacySecurity: true` in the AWF config if host-access is required — `network.isolation: false` alone is insufficient because `applySecurityMode()` forces isolation back on and clears `enableHostAccess` in default strict mode (`src/commands/validators/security-mode.ts:37-74`). B16 / github/gh-aw-firewall#6649 — AWF rejects `--mount` entries whose host path contains an unexpanded `${VAR}` reference (for example `${TERRAFORM_CLI_PATH}/terraform:/usr/local/bin/terraform`) with a "host path must be absolute" error. The gh-aw compiler wraps `sandbox.agent.mounts` specs containing `${}` in single quotes in the generated shell invocation; Bash single quotes prevent variable substitution, so the literal string reaches AWF's volume validator instead of the resolved path. **Unresolved** (github/gh-aw-firewall#6649 open). Workaround: replace `${VAR}` references with their literal values in `sandbox.agent.mounts`, or stage the path into a wrapper script that expands before calling AWF. diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index ab20a3edd..35bb778d9 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -51,9 +51,9 @@ Establish these facts before matching a failure mode: | B10 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` × 3 directories, each taking exactly ~30 s; total ~90 s wasted; cascading `EACCES: permission denied` / `Failed to remove chroot home directory after permission repair` | `fixArtifactPermissionsForRootless()` builds compound `tag@digest` image refs (e.g. `agent:0.27.22@sha256:55f065...`) for its `docker run --pull never` repair container. Despite `--pull never`, Docker still attempts registry manifest verification for compound refs; when GHCR credentials are unavailable (cleaned in an earlier workflow step or expired) the verification TCP-connects and times out (~30 s per directory). | **Fixed in AWF (PR github/gh-aw-firewall#6025)**: `resolvePermFixerImageRef()` now strips the `@sha256:...` digest and uses a tag-only ref. `--pull never` with a tag-only ref skips all registry I/O. Upgrade to AWF version that includes github/gh-aw-firewall#6025. Workaround (older AWF): ensure GHCR credentials are available until after the AWF cleanup step. **Additional fix (PRs github/gh-aw-firewall#6342 / github/gh-aw-firewall#6356, merged 2026-07-18):** Even with a tag-only image ref, `docker run` was missing `--entrypoint sh`, causing AWF's `entrypoint.sh` to execute in place of the repair command. `entrypoint.sh` waits ~30 s for an iptables-init container that never starts in this context, then fails. The fix adds `--entrypoint sh` and passes the `chown/chmod` command via `-c`; stdout is now captured alongside stderr so timeout messages surfaced on stdout are not silently swallowed. | `[WARN] Rootless artifact permission repair failed` messages each followed by exactly ~30 s delay in the workflow log; `docker pull --dry-run agent:...@sha256:...` times out while `docker inspect agent:...` succeeds | github/gh-aw-firewall#6025, github/gh-aw-firewall#6342, github/gh-aw-firewall#6356 | | B11 | AWF exits with code 1 and logs show `[WARN] Rootless artifact permission repair failed ... (exit 1)` plus cleanup warnings, but no actionable stderr detail from the repair container | `fixArtifactPermissionsForRootless()` previously discarded repair-container stderr, so the warning hid the real failure context. The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it. | **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured in the warning, and chroot-home cleanup noise is reduced by downgrading that log from `warn` to `debug`. Treat this mode as diagnostic-opacity around a pre-existing non-zero command exit, not cleanup-driven exit-code mutation. | Compare agent-command logs with `Command completed with exit code: `; on affected versions, repair warnings lack stderr context. On AWF including github/gh-aw-firewall#6072, the warning includes stderr detail for root-cause triage. | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 | | B12 | `getaddrinfo EAI_AGAIN ` or `ENOTFOUND ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` on ARC/DinD, and DinD containers cannot resolve the Kubernetes Service name even after the ordering fix | ARC/DinD DNS isolation: the DinD Docker network does not forward DNS to the cluster resolver, so Kubernetes Service names such as `awmg-cli-proxy` are unreachable from AWF containers even though the service exists. | **Diagnosed in AWF (PR github/gh-aw-firewall#6328, merged 2026-07-17):** `detectDnsResolutionFailure()` scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with concrete remediation. Address the DIFC proxy by IP, or configure `dockerd --dns ` so DinD containers can reach the cluster DNS server. | `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers; compare with the augmented startup error from `detectDnsResolutionFailure()` | github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | -| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the Firewall Issue Dispatcher block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 | +| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. **Open sub-issue (github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658):** After github/gh-aw-firewall#6473, topology peer hostnames (e.g. `awmg-mcpg`) receive runtime Squid allow rules and traffic is permitted at the network level. However, the Firewall Issue Dispatcher **audit/block report** is generated from the policy manifest rather than the live Squid config; the manifest does not yet include topology-peer entries, so the block report incorrectly flags `awmg-mcpg` as denied. This is a **reporting false positive** — actual traffic is not blocked. The report also incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Tracked in github/gh-aw-firewall#6658. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .` after github/gh-aw-firewall#6473; if the Firewall Issue Dispatcher block report flags a topology peer after github/gh-aw-firewall#6473, treat it as a reporting false positive (manifest attribution gap; tracked in github/gh-aw-firewall#6658) | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652, github/gh-aw-firewall#6658 | | B14 | After upgrading to AWF including the centralized mount-policy refactor (PR github/gh-aw-firewall#6339), the Copilot CLI agent exits immediately (exit code 1, ~0.5 s, zero stdout/stderr); no network calls are made; the issue appears on Docker and gVisor runtimes but not sbx (which moves files aside rather than overlay-masking them) | PR github/gh-aw-firewall#6339 added `~/.copilot/config.json` to the credential **deny list** with reason *"Copilot CLI persisted auth token"*. This is incorrect: `config.json` contains only experiment flags, first-launch timestamps, and plugin metadata — no token. AWF masks denied files with a **read-only** `/dev/null` bind-mount overlay. The Copilot CLI reads **and atomically rewrites** `config.json` via temp-file + rename at startup; a rename cannot target a bind-mounted path and the overlay is read-only, so the CLI crashes silently. | **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `~/.copilot/config.json` is removed from the credential deny list; the file is bind-mounted read-write as part of the `.copilot` tool subdir (pre-#6339 behaviour). Upgrade to the AWF version including github/gh-aw-firewall#6374. Workaround (older AWF): pass `--keep-containers` and inspect the agent container for `EROFS` or rename errors; downgrade AWF past github/gh-aw-firewall#6339 if unpatched. | `awf ... copilot agent --version` — if it exits 1 with no output in ~0.5 s, confirm AWF version includes github/gh-aw-firewall#6374; inspect agent container logs for `EROFS`/`rename` errors on `~/.copilot/config.json` | github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 | -| B15 | AWF startup fails with `--network-isolation is not yet supported with --enable-host-access`; happens when `localhost` is in the domain allowlist because the gh-aw compiler auto-enables `--enable-host-access` and also emits `network.isolation: true` + `topologyAttach` | In `src/cli.ts`, `--network-isolation` and `--enable-host-access` are mutually exclusive with no overlap support. The gh-aw compiler (v0.82.x+) does not detect the conflict: it emits `isolation:true` + `topologyAttach` unconditionally while also auto-enabling `--enable-host-access` when `localhost` is in the allowlist. The compiled workflow always fails at runtime. | **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required. | `awf --network-isolation --enable-host-access ...` — reproduces immediately with `not yet supported` error; check compiler output for both flags being emitted together when `localhost` is in allowlist | github/gh-aw-firewall#6651 | +| B15 | AWF startup fails with `--network-isolation is not yet supported with --enable-host-access`; happens when `localhost` is in the domain allowlist because the gh-aw compiler auto-enables `--enable-host-access` and also emits `network.isolation: true` + `topologyAttach` | In `src/cli.ts`, `--network-isolation` and `--enable-host-access` are mutually exclusive with no overlap support. The gh-aw compiler (v0.82.x+) does not detect the conflict: it emits `isolation:true` + `topologyAttach` unconditionally while also auto-enabling `--enable-host-access` when `localhost` is in the allowlist. The compiled workflow always fails at runtime. | **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set both `network.isolation: false` **and** `security.legacySecurity: true` in the AWF config if host-access is required — `network.isolation: false` alone is insufficient because `applySecurityMode()` forces isolation back on and clears `enableHostAccess` in default strict mode (`src/commands/validators/security-mode.ts:37-74`). | `awf --network-isolation --enable-host-access ...` — reproduces immediately with `not yet supported` error; check compiler output for both flags being emitted together when `localhost` is in allowlist | github/gh-aw-firewall#6651 | | B16 | AWF rejects a `--mount` entry with a non-absolute host path error; the mount spec was defined as `${TERRAFORM_CLI_PATH}/terraform:...` (or similar env-var pattern) but the literal string `${TERRAFORM_CLI_PATH}` reaches AWF's volume validator | The gh-aw compiler wraps `sandbox.agent.mounts` specs containing `${}` references in single quotes in the generated shell invocation. Bash single quotes prevent variable substitution, so the literal variable reference string (e.g. `${TERRAFORM_CLI_PATH}`) reaches AWF's volume validation instead of the resolved absolute path. | **Unresolved** (github/gh-aw-firewall#6649 open). Workaround: replace `${VAR}` references with their literal values in `sandbox.agent.mounts`, or stage the path into a wrapper script that expands before calling AWF. | Inspect the generated `awf` invocation in the compiled lock file — if mount specs appear in single quotes containing `${...}`, the bug is present. Running `awf ... --mount '${VAR}/path:/dest'` reproduces the rejection. | github/gh-aw-firewall#6649 | ## Category C — GHES / GHEC / `ghe.com` @@ -123,7 +123,7 @@ Establish these facts before matching a failure mode: | Copilot CLI exits immediately (exit 1, ~0.5 s, zero stdout/stderr) after AWF upgrade; works in sbx but not Docker/gVisor | B14 (`.copilot/config.json` masked as credential; fixed in github/gh-aw-firewall#6374) | | `copilot: command not found` inside sbx microVM when binary is at `~/.local/bin/copilot` | D10 (`bash -lc` login init resets injected PATH; fixed by command wrapper `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) | | `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) | -| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap) | +| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) | | `--network-isolation is not yet supported with --enable-host-access` | B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; root cause unresolved in github/gh-aw-firewall#6651) | | AWF rejects `--mount` with "host path must be absolute" (or similar) and the path visibly contains `${VAR_NAME}` unexpanded | B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; root cause unresolved in github/gh-aw-firewall#6649) |