test(e2e): add reusable QEMU infrastructure for E2E tests by drew · Pull Request #2471 · NVIDIA/OpenShell

drew · 2026-07-24T17:55:59Z

Summary

Add reusable Nix and QEMU infrastructure for running OpenShell tests in disposable Linux VMs. The harness provides pinned native-architecture cloud guests, composable runtime and security configurations, artifact installation, predictable cleanup, and retained diagnostics for failures.

This is the base infrastructure PR for the VM-backed release and E2E test integrations in the child PRs below.

Stack

This is the parent PR.

ci(release): run Linux release canaries in QEMU VMs #2472 — run Linux release canaries in QEMU VMs
test(e2e): add VM-backed E2E suite runner #2473 — run E2E suites through the QEMU VM infrastructure

Related Issue

feat: E2E and release testing improvements #2527
Prototype alternative to feat(e2e): use xtask for host Linux workflows #2254.

Changes

Add pinned Ubuntu 24.04, Fedora 44, CentOS Stream 10, and Rocky Linux 9 cloud-image definitions for ARM64 and AMD64
Add a nix run .#test-vm interface with disposable QCOW2 overlays, cloud-init SSH bootstrap, hardware acceleration where available, cleanup, and retained debug state
Add composable Ansible configurations for Docker, rootless Podman, and SELinux enforcing mode
Support repeatable package installation and direct executable copying into guests
Add automatic KVM-to-TCG fallback for Linux hosts without accessible hardware virtualization
Add dpkg to the Nix development shell and keep Debian package creation artifact-only
Document requirements, supported combinations, package workflows, lifecycle, and current limitations

Testing

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture and VM usage documentation updated

copy-pr-bot · 2026-07-24T17:56:02Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

copy-pr-bot · 2026-07-28T18:54:28Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR is project-valid as reusable QEMU/Nix test infrastructure for the release and E2E stack.
Head SHA: 52cb5e1b90ababfdf89a4ee95a37bb111509263f

Review findings:

Warning: the OCI cache path pulls and boots a full guest disk from a mutable repository reference. Metadata and disk checksums protect transfer integrity, but not provenance. Before wiring this into trusted CI, require an org-controlled immutable registry policy or artifact signing/verification; otherwise the cache repository becomes a supply-chain execution surface.
Three line-specific warnings are inline.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the new workflow.

Next state: gator:in-review

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test guest infrastructure for release and E2E follow-on work.
Head SHA: 49ae902c28f3c514535434e65bc4dc2c640be5da

Review findings:

Warning: package verification coverage regressed when the Debian package smoke-test block was removed before an equivalent VM-backed CI replacement exists.
Warning: OCI prepared-disk cache pulls still verify integrity but not publisher provenance; trusted CI use needs an immutable org-controlled reference or signature/manifest verification.
Suggestion: nix/test-guest/run.sh should use the same explicit option-value validation pattern as cache.sh for a cleaner CLI failure mode.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the workflow.

Checks: DCO and gate-status publication passed; Branch Checks and Helm Lint are still pending on /ok to test mirror validation. I am not triggering CI while review warnings remain unresolved.

Next state: gator:in-review

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew · 2026-07-28T20:46:43Z

/ok to test 9f24004

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test guest infrastructure for release and E2E follow-on work.
Head SHA: 9f240044655387561c88b4b305e8770f88042765

Review findings:

Warning: the SSH control socket path can exceed Unix socket path limits during cache builds because TMPDIR may be a deep cache staging path.
Warning: local prepared-disk cache hits still trust existing metadata without revalidating the disk checksum or QCOW2 structure.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the workflow.

Checks: DCO and gate-status publication passed; Branch Checks and Helm Lint are still pending on /ok to test mirror validation. I am not triggering CI while review warnings remain unresolved.

Next state: gator:in-review

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test guest infrastructure for release and E2E follow-on work.
Head SHA: 7f365efb813777575f291f6e48230913aa287303

Review findings:

Warning: removing the Debian package smoke block drops existing package verification coverage before the VM-backed replacement lands.
Suggestion: the runner option parser should validate required option values before shift 2 for cleaner CLI failures.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the workflow.

Checks: DCO and gate-status publication passed; Branch Checks and Helm Lint are still pending on /ok to test mirror validation. I am not moving to pipeline watch while the review warning remains unresolved.

Next state: gator:in-review

drew · 2026-07-28T21:58:52Z

-# Smoke tests
-# ---------------------------------------------------------------------------
-
-extract_dir="${tmpdir}/extract"


gator-agent

Warning: Removing the package smoke block drops existing CI coverage before the VM-backed replacement lands. Keep the extracted --version checks and systemd-analyze --user verify here, or add an equivalent check to the current Debian package workflow in this PR.

drew · 2026-07-28T21:58:52Z

+while [ "$#" -gt 0 ]; do
+	case "$1" in
+	--distro)
+		distro=${2:-}


gator-agent

Suggestion: Options that require values still do shift 2 without validation, so --distro or --install at EOF exits with a raw shell error. Mirror cache.sh’s require_value "$@" helper for --distro, --with, --install, --copy, and --ssh-port.

drew mentioned this pull request Jul 24, 2026

ci(release): run Linux release canaries in QEMU VMs #2472

Draft

10 tasks

drew changed the title ~~test(vm): add composable QEMU test guests~~ test(vm): add reusable QEMU infrastructure for E2E tests Jul 24, 2026

drew mentioned this pull request Jul 24, 2026

test(e2e): add VM-backed E2E suite runner #2473

Draft

13 tasks

drew changed the title ~~test(vm): add reusable QEMU infrastructure for E2E tests~~ test(e2e): add reusable QEMU infrastructure for E2E tests Jul 24, 2026

drew force-pushed the drew/e2e-vm-example branch from a9988d0 to 41f2720 Compare July 24, 2026 23:17

drew added 6 commits July 28, 2026 12:28

test(vm): add composable QEMU test guests

9de27e5

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

docs(vm): describe test VM directory structure

2a9d51d

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

refactor(vm): replace shell catalog functions

dc2a466

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

test(vm): add Fedora release guest support

59aa8d9

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

test(vm): enable rootless Podman socket

e2351e2

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

feat(test-guest): add OCI-backed image caching

52cb5e1

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew force-pushed the drew/e2e-vm-example branch from 4f528ef to 52cb5e1 Compare July 28, 2026 19:29

drew marked this pull request as ready for review July 28, 2026 19:47

drew requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners July 28, 2026 19:47

drew commented Jul 28, 2026

View reviewed changes

Comment thread nix/test-guest/run.sh

Comment thread nix/test-guest/configuration/docker.yml

Comment thread tasks/scripts/package-deb.sh

drew added the gator:in-review Gator is reviewing or awaiting PR review feedback label Jul 28, 2026

perf(test-guest): accelerate cached guest startup

eebc077

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew mentioned this pull request Jul 28, 2026

feat: E2E and release testing improvements #2527

Open

5 tasks

fix(test-guest): address review feedback

49ae902

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew commented Jul 28, 2026

View reviewed changes

Comment thread tasks/scripts/package-deb.sh

Comment thread nix/test-guest/cache.sh Outdated

fix(test-guest): verify OCI cache provenance

9f24004

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew force-pushed the drew/e2e-vm-example branch from 039537a to 9f24004 Compare July 28, 2026 20:42

drew commented Jul 28, 2026

View reviewed changes

Comment thread nix/test-guest/run.sh Outdated

Comment thread nix/test-guest/cache-lib.sh

fix(test-guest): harden cached guest reuse

7f365ef

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew added 2 commits July 28, 2026 14:54

fix(test-guest): refresh runtime setup state

7bd0790

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

feat(test-guest): support E2E runner inputs

418b260

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew commented Jul 28, 2026

View reviewed changes

Uh oh!

Conversation

drew commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Jul 24, 2026

Uh oh!

copy-pr-bot Bot commented Jul 28, 2026

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

Uh oh!

drew commented Jul 28, 2026

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

drew Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

drew Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

drew commented Jul 24, 2026 •

edited

Loading