Skip to content

docs: Runner Doctor 2026-07-27 scan — B13 update + B15/B16 new failure modes - #6659

Merged
lpcox merged 3 commits into
mainfrom
copilot/update-runner-doctor-b13
Jul 28, 2026
Merged

docs: Runner Doctor 2026-07-27 scan — B13 update + B15/B16 new failure modes#6659
lpcox merged 3 commits into
mainfrom
copilot/update-runner-doctor-b13

Conversation

Copilot AI commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Applies the 2026-07-27 Runner Doctor scan findings to all three mirrored catalog files (shared/self-hosted-failure-modes.md, self-hosted-runner-doctor.md, .github/agents/self-hosted-runner-doctor.md).

B13 — remaining Squid ACL gap (open sub-issue #6652)

  • Appends an Open sub-issue note to the Fix/flag column: topologyAttach MCP gateway hostname (awmg-mcpg) is still blocked by Squid in some configurations even after Auto-allow topology-attached container hostnames in Squid ACL #6473; the block report incorrectly recommends adding awmgmcpg to network.allowed, which fails schema validation
  • Adds #6652 to Citations

B15 — new failure mode (#6651)

--network-isolation + --enable-host-access mutual-exclusion conflict: the gh-aw compiler (v0.82.x+) auto-enables --enable-host-access when localhost is in the allowlist while also unconditionally emitting network.isolation: true + topologyAttach; AWF rejects the combination at startup. Unresolved.

B16 — new failure mode (#6649)

${VAR} in sandbox.agent.mounts reaches AWF's volume validator unexpanded: the compiler wraps specs containing ${} in single quotes, preventing shell expansion. AWF then rejects the non-absolute path. Unresolved.

Each failure mode is added to: the category table, the error-string quick-lookup table, and the known-unresolved list — in all three mirrored files.

Copilot AI changed the title [WIP] Update Runner Doctor for new failure modes and B13 fix docs: Runner Doctor 2026-07-27 scan — B13 update + B15/B16 new failure modes Jul 27, 2026
Copilot finished work on behalf of lpcox July 27, 2026 20:04
Copilot AI requested a review from lpcox July 27, 2026 20:04
@lpcox
lpcox marked this pull request as ready for review July 27, 2026 22:50
Copilot AI review requested due to automatic review settings July 27, 2026 22:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates Runner Doctor guidance with new and revised AWF failure modes.

Changes:

  • Revises B13 topology-peer diagnostics.
  • Adds unresolved B15 and B16 modes.
  • Synchronizes all three catalog copies.
Show a summary per file
File Description
.github/workflows/shared/self-hosted-failure-modes.md Updates the shared catalog.
.github/workflows/self-hosted-runner-doctor.md Updates workflow diagnostics.
.github/agents/self-hosted-runner-doctor.md Updates the portable catalog mirror.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (5)

.github/workflows/self-hosted-runner-doctor.md:149

  • The appended root cause is stale: #6473 already supplies topologyPeers to Squid and generates pre-Safe_ports allow rules (src/config-writer.ts:335-340, src/squid/config-sections.ts:150-173). According to follow-up PR #6658, only the policy manifest omitted these rules, causing audit/block-report misattribution. Replace the claim that Squid still blocks the gateway with the manifest/reporting gap and its fix status.
B13 / github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 — In `--network-isolation` mode, MCP tool calls to topology-attached peers or to a `difcProxyHost` were failing because those hosts were not included in `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** topology peer hostnames are added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of `:port`) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist. Upgrade to AWF including all three PRs. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation.

.github/agents/self-hosted-runner-doctor.md:120

  • The new #6652 explanation is inaccurate: topology peers are already allowed by generated Squid rules before Safe_ports (src/config-writer.ts:335-340, src/squid/config-sections.ts:150-173). Follow-up PR #6658 shows the missing rule is in the policy manifest used by audit/block reporting. Recast this as an audit-attribution gap rather than an actual runtime block.
B13 / github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 — In `--network-isolation` mode, MCP tool calls to topology-attached peers or to a `difcProxyHost` were failing because those hosts were not included in `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** topology peer hostnames are added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of `:port`) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist. Upgrade to AWF including all three PRs. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation.

.github/agents/self-hosted-runner-doctor.md:221

  • This catalog row says Squid still blocks awmg-mcpg, but the runtime config already passes topology peers and emits explicit allow rules (src/config-writer.ts:335-340, src/squid/config-sections.ts:150-173). PR #6658 identifies the real defect as omission from the policy manifest, which makes audit reports recommend an invalid allowlist entry. Update the row to distinguish working traffic from the reporting false positive.
| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the Firewall Issue Dispatcher block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .<topology-host>` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 |

.github/agents/self-hosted-runner-doctor.md:293

  • Calling this a remaining Squid ACL gap will misroute diagnosis. The runtime Squid rules already allow topology peers (src/config-writer.ts:335-340, src/squid/config-sections.ts:150-173); PR #6658 shows that only the audit policy manifest is missing the equivalent rule. Update this lookup to point to an audit/block-report false positive.
| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap) |

.github/agents/self-hosted-runner-doctor.md:223

  • The host-access workaround is incomplete: under default strict mode, AWF ignores network.isolation: false and disables enableHostAccess (src/commands/validators/security-mode.ts:37-74). It must also set security.legacySecurity: true; otherwise following this row does not restore host access.
| B15 | AWF startup fails with `--network-isolation is not yet supported with --enable-host-access`; happens when `localhost` is in the domain allowlist because the gh-aw compiler auto-enables `--enable-host-access` and also emits `network.isolation: true` + `topologyAttach` | In `src/cli.ts`, `--network-isolation` and `--enable-host-access` are mutually exclusive with no overlap support. The gh-aw compiler (v0.82.x+) does not detect the conflict: it emits `isolation:true` + `topologyAttach` unconditionally while also auto-enabling `--enable-host-access` when `localhost` is in the allowlist. The compiled workflow always fails at runtime. | **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required. | `awf --network-isolation --enable-host-access ...` — reproduces immediately with `not yet supported` error; check compiler output for both flags being emitted together when `localhost` is in allowlist | github/gh-aw-firewall#6651 |
  • Files reviewed: 3/3 changed files
  • Comments generated: 7
  • Review effort level: Medium

| B11 | AWF exits with code 1 and logs show `[WARN] Rootless artifact permission repair failed ... (exit 1)` plus cleanup warnings, but no actionable stderr detail from the repair container | `fixArtifactPermissionsForRootless()` previously discarded repair-container stderr, so the warning hid the real failure context. The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it. | **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured in the warning, and chroot-home cleanup noise is reduced by downgrading that log from `warn` to `debug`. Treat this mode as diagnostic-opacity around a pre-existing non-zero command exit, not cleanup-driven exit-code mutation. | Compare agent-command logs with `Command completed with exit code: <n>`; on affected versions, repair warnings lack stderr context. On AWF including github/gh-aw-firewall#6072, the warning includes stderr detail for root-cause triage. | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 |
| B12 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` or `ENOTFOUND <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` on ARC/DinD, and DinD containers cannot resolve the Kubernetes Service name even after the ordering fix | ARC/DinD DNS isolation: the DinD Docker network does not forward DNS to the cluster resolver, so Kubernetes Service names such as `awmg-cli-proxy` are unreachable from AWF containers even though the service exists. | **Diagnosed in AWF (PR github/gh-aw-firewall#6328, merged 2026-07-17):** `detectDnsResolutionFailure()` scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with concrete remediation. Address the DIFC proxy by IP, or configure `dockerd --dns <cluster-dns-ip>` so DinD containers can reach the cluster DNS server. | `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers; compare with the augmented startup error from `detectDnsResolutionFailure()` | github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 |
| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .<topology-host>` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473 |
| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the Firewall Issue Dispatcher block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .<topology-host>` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 |
| `copilot: command not found` inside sbx microVM when binary is at `~/.local/bin/copilot` | D10 (`bash -lc` login init resets injected PATH; fixed by command wrapper `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) |
| `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) |
| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438) |
| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap) |
| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .<topology-host>` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473 |
| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. **Open sub-issue (github/gh-aw-firewall#6652):** `topologyAttach` MCP gateway hostname (`awmg-mcpg`) is still blocked by Squid in some configurations after github/gh-aw-firewall#6473; the Firewall Issue Dispatcher block report incorrectly suggests adding `awmgmcpg` to `network.allowed`, which fails schema validation. Root cause under investigation. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .<topology-host>` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473, github/gh-aw-firewall#6652 |
| B14 | After upgrading to AWF including the centralized mount-policy refactor (PR github/gh-aw-firewall#6339), the Copilot CLI agent exits immediately (exit code 1, ~0.5 s, zero stdout/stderr); no network calls are made; the issue appears on Docker and gVisor runtimes but not sbx (which moves files aside rather than overlay-masking them) | PR github/gh-aw-firewall#6339 added `~/.copilot/config.json` to the credential **deny list** with reason *"Copilot CLI persisted auth token"*. This is incorrect: `config.json` contains only experiment flags, first-launch timestamps, and plugin metadata — no token. AWF masks denied files with a **read-only** `/dev/null` bind-mount overlay. The Copilot CLI reads **and atomically rewrites** `config.json` via temp-file + rename at startup; a rename cannot target a bind-mounted path and the overlay is read-only, so the CLI crashes silently. | **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `~/.copilot/config.json` is removed from the credential deny list; the file is bind-mounted read-write as part of the `.copilot` tool subdir (pre-#6339 behaviour). Upgrade to the AWF version including github/gh-aw-firewall#6374. Workaround (older AWF): pass `--keep-containers` and inspect the agent container for `EROFS` or rename errors; downgrade AWF past github/gh-aw-firewall#6339 if unpatched. | `awf ... copilot agent --version` — if it exits 1 with no output in ~0.5 s, confirm AWF version includes github/gh-aw-firewall#6374; inspect agent container logs for `EROFS`/`rename` errors on `~/.copilot/config.json` | github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 |
| B15 | AWF startup fails with `--network-isolation is not yet supported with --enable-host-access`; happens when `localhost` is in the domain allowlist because the gh-aw compiler auto-enables `--enable-host-access` and also emits `network.isolation: true` + `topologyAttach` | In `src/cli.ts`, `--network-isolation` and `--enable-host-access` are mutually exclusive with no overlap support. The gh-aw compiler (v0.82.x+) does not detect the conflict: it emits `isolation:true` + `topologyAttach` unconditionally while also auto-enabling `--enable-host-access` when `localhost` is in the allowlist. The compiled workflow always fails at runtime. | **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required. | `awf --network-isolation --enable-host-access ...` — reproduces immediately with `not yet supported` error; check compiler output for both flags being emitted together when `localhost` is in allowlist | github/gh-aw-firewall#6651 |
- `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407)
- `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558)
- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438)
- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap)

B14 / github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 — After the centralized mount-policy refactor (github/gh-aw-firewall#6339), `~/.copilot/config.json` was incorrectly added to the credential deny list, causing AWF to overlay it with a read-only `/dev/null` bind mount. The Copilot CLI atomically rewrites this file at startup via temp-file + rename; the read-only overlay makes the rename fail and the CLI exits silently (exit 1, ~0.5 s). **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `config.json` is removed from the deny list and restored to read-write. Workaround (older AWF): downgrade past github/gh-aw-firewall#6339 or pass `--keep-containers` and inspect for `EROFS`/rename errors.

B15 / github/gh-aw-firewall#6651 — AWF startup fails with `--network-isolation is not yet supported with --enable-host-access` when `localhost` is in the domain allowlist. The gh-aw compiler (v0.82.x+) auto-enables `--enable-host-access` when `localhost` is in the allowlist and also emits `network.isolation: true` + `topologyAttach` unconditionally; the two flags are mutually exclusive in `src/cli.ts` with no overlap support. **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required.
- `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407)
- `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558)
- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438)
- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; open sub-issue github/gh-aw-firewall#6652 for remaining Squid ACL gap)

B14 / github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 — After the centralized mount-policy refactor (github/gh-aw-firewall#6339), `~/.copilot/config.json` was incorrectly added to the credential deny list, causing AWF to overlay it with a read-only `/dev/null` bind mount. The Copilot CLI atomically rewrites this file at startup via temp-file + rename; the read-only overlay makes the rename fail and the CLI exits silently (exit 1, ~0.5 s). **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `config.json` is removed from the deny list and restored to read-write. Workaround (older AWF): downgrade past github/gh-aw-firewall#6339 or pass `--keep-containers` and inspect for `EROFS`/rename errors.

B15 / github/gh-aw-firewall#6651 — AWF startup fails with `--network-isolation is not yet supported with --enable-host-access` when `localhost` is in the domain allowlist. The gh-aw compiler (v0.82.x+) auto-enables `--enable-host-access` when `localhost` is in the allowlist and also emits `network.isolation: true` + `topologyAttach` unconditionally; the two flags are mutually exclusive in `src/cli.ts` with no overlap support. **Unresolved** (github/gh-aw-firewall#6651 open). Short-term workaround: remove `localhost` from the domain allowlist if topology/isolation is required; or set `network.isolation: false` in the AWF config if host-access is required.
@lpcox

lpcox commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

@copilot address review feedback

- B13: reword 'remaining Squid ACL gap' as audit/policy-manifest
  reporting false positive in all 3 files (category table, quick-lookup,
  and condensed-description sections); add #6658 as the tracking PR
- B15: add security.legacySecurity: true to the network.isolation: false
  workaround in all 3 files — applySecurityMode() forces isolation back
  on and clears enableHostAccess in default strict mode without it

Copilot AI commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address review feedback

Addressed in commit docs: fix B13 diagnosis and B15 workaround per review feedback:

B13 — Removed all "remaining Squid ACL gap" / "Root cause under investigation" language from all three mirrored files (category table, quick-lookup table, and condensed-description section). The open sub-issue is now described as an audit/policy-manifest reporting false positive: topology peers receive runtime Squid allow rules after #6473 and traffic is not blocked; the Firewall Issue Dispatcher block report is generated from the policy manifest (not the live Squid config) and doesn't include topology-peer entries, causing awmg-mcpg to appear denied when it isn't. Added #6658 as the tracking reference alongside #6652 in all affected locations.

B15 — Updated the workaround in all three files to require both network.isolation: false and security.legacySecurity: true, with a note explaining that network.isolation: false alone is insufficient because applySecurityMode() forces isolation back on and clears enableHostAccess in default strict mode (src/commands/validators/security-mode.ts:37-74).

Copilot finished work on behalf of lpcox July 27, 2026 23:17
@github-actions

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini completed. All facets verified. 💎

GitHub connectivity: ❌ (Exit Code 7/6)

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Contribution Check failed. Please review the logs for details.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

@github-actions

Copy link
Copy Markdown
Contributor

🔥 Smoke Test Results — PASS

Test Result
GitHub MCP connectivity
GitHub.com HTTP ✅ 200
File write/read

Overall: PASS · @lpcox

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🔥 Smoke Test: Copilot Network Isolation

@lpcox

EGRESS_RESULT allow=pass deny=pass

  • ✅ Allowed domain (api.github.com) — HTTP 200
  • ✅ Blocked domain (example.com) — proxy 403, curl blocked

Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions github-actions Bot added the smoke-copilot-network-isolation Copilot network-isolation egress smoke test label Jul 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Overall Result: ✅ PASS

Check Status
API Status ✅ PASS
GH Check ✅ PASS
File Status ✅ PASS

Generated by Smoke Claude for #6659 · haiku45 · 36.5 AIC · ⊞ 3.3K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) Mode

✅ GitHub MCP connectivity (list PRs)
✅ GitHub.com connectivity (HTTP 200)
✅ File I/O test (read /tmp/gh-aw/agent_cli_start_ms.txt)
✅ BYOK inference working (direct mode via api-proxy → api.githubcopilot.com)

Status: PASS

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) with api-proxy sidecar credential injection.

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test Results

Overall Status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.13 Python 3.12.13 ✅ YES
Node.js v24.18.0 v22.23.1 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Overall: ❌ FAILED — Node.js version mismatch (host v24.18.0 vs chroot v22.23.1).

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🔭 Smoke Test: API Proxy OpenTelemetry Tracing

Scenario Status Notes
S1: Module Loading otel.js loads; exports startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled
S2: Test Suite otel.test.js + otel-fanout.test.js present with coverage for span creation, token attrs, graceful degradation
S3: Env Var Forwarding api-proxy-env-config.test.ts confirms GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID forwarded
S4: Token Tracker Integration onUsage callback present in token-tracker-http.js (line 406) as OTEL hook point
S5: OTEL Diagnostics Graceful fallback: spans write to /var/log/api-proxy/otel.jsonl when no OTLP endpoint configured

All 5 scenarios pass. OTEL tracing integration is fully wired: module loads, spans created per request with GenAI semantic conventions, token usage hooked via onUsage, env vars forwarded from AWF host → api-proxy container, and export routes through Squid proxy.

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy -> Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

  • GitHub MCP connectivity: ✅
  • GitHub.com connectivity: ✅
  • File write/read: ✅
  • BYOK inference: ✅

Overall: PASS

@lpcox

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test Results — Services Connectivity

Check Result
Redis PING ❌ Name resolution failed for host.docker.internal
PostgreSQL pg_isready ❌ No response
PostgreSQL SELECT 1 ❌ Name resolution failed

Overall: FAILhost.docker.internal is not resolvable in this environment. Service containers are unreachable.

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color passed ✅ PASS
Go env passed ✅ PASS
Go uuid passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx passed ✅ PASS
Node.js execa passed ✅ PASS
Node.js p-limit passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Note (Java): The default ~/.m2 directory was owned by root (pre-existing environment condition). Maven was invoked with -Dmaven.repo.local=/tmp/gh-aw/agent/m2repo to use a writable path; all tests passed successfully.

Generated by Build Test Suite for #6659 · sonnet46 · 61.7 AIC · ⊞ 8.3K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test Results — Docker Sbx

Test Result
GitHub MCP connectivity ❌ Filtered by secrecy policy
GitHub.com HTTP connectivity ⚠️ Pre-step output not substituted
File write/read ⚠️ Pre-step output not substituted

Overall: ⚠️ INCONCLUSIVE — workflow template variables were not expanded (pre-step outputs unavailable). Tests could not be fully validated.

📰 BREAKING: Report filed by Smoke Docker Sbx
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

@lpcox Smoke Test:

  • GitHub MCP: ❌ (MCP error)
  • GitHub.com: ❌ (HTTP code !=200)
  • File I/O: ✅
  • BYOK Inference: ✅
    Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (o4-mini-aw)
    Overall: FAIL

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

✅ fix: add topologyPeers to policy manifest for accurate audit attribution
✅ fix: firewall-issue-dispatcher always skipping activation
❌ safeinputs-gh query unavailable
✅ Playwright GitHub title check
✅ file write/readback
❌ discussion query unavailable
✅ npm ci && npm run build
Overall: FAIL

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@lpcox
lpcox merged commit 028e671 into main Jul 28, 2026
127 of 129 checks passed
@lpcox
lpcox deleted the copilot/update-runner-doctor-b13 branch July 28, 2026 00:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🩺 Runner Doctor UpdateRunner Doctor: 2026-07-27 scan — B13 update + two new failure modes (B15, B16)

3 participants