ci(release): run Linux release canaries in QEMU VMs

.agents/skills/test-release-canary/SKILL.md

@@ -12,11 +12,14 @@ The Release Canary (`.github/workflows/release-canary.yml`) smoke-tests the arti
 | Job | Runner | Verifies |
 |---|---|---|
 | `macos` | `macos-latest-xlarge` | `install.sh` resolves the Homebrew formula, brew installs the cask, and `openshell status` reaches the brew-services–backed local gateway with the VM driver. |
-| `ubuntu` | `ubuntu-latest` | `install.sh` installs the Debian package, the post-install systemd user service starts, and `openshell status` reaches the local gateway with the Docker driver. |
-| `fedora` | `fedora:latest` container | `install.sh` installs the RPM packages, the local gateway starts under Podman, and `openshell status` succeeds. |
+| `linux-vm` (Ubuntu) | `ubuntu-24.04` / `ubuntu-24.04-arm` | The Nix VM harness boots Ubuntu 24.04 on AMD64 and ARM64, configures Docker, installs the rolling `dev` Debian packages, and verifies the local gateway. |
+| `linux-vm` (Fedora) | `ubuntu-24.04` / `ubuntu-24.04-arm` | The Nix VM harness boots Fedora 44 on AMD64 and ARM64, configures rootless Podman with enforcing SELinux, installs the rolling `dev` RPM packages, and verifies the local gateway. |
+| `ubuntu-snap` | `ubuntu-latest` | The Snap produced by the triggering Release Dev run installs with its required interfaces and reaches the Docker-backed gateway. |
 | `kubernetes` | `ubuntu-latest` + kind | `helm install oci://ghcr.io/nvidia/openshell/helm-chart --version 0.0.0-dev` succeeds in a kind cluster, the gateway pod becomes Ready, port-forward exposes 8080, and the released CLI registers the in-cluster gateway and runs `openshell status` against it. |
 
-`install.sh` defaults to the *latest tagged* release — the canary is therefore checking that the most recent public release still installs, not the just-published `dev` build. The `kubernetes` job is the exception: it pins to `0.0.0-dev` chart + `:dev` images.
+The Linux VM matrix sets `OPENSHELL_VERSION=dev`, so it tests the packages published by Release Dev. The Snap job downloads artifacts from the triggering run, and Kubernetes pins the `0.0.0-dev` chart and `:dev` images. The macOS job retains the installer's default latest-tagged release behavior.
+
+On Linux, the VM harness uses KVM when `/dev/kvm` is readable and writable. It falls back automatically to native-architecture QEMU TCG when KVM is unavailable, which is expected on public ARM64 runners. The ARM64 fallback switches from the default firmware to QEMU's bundled EDK2 firmware, which is validated under TCG. Normal canaries do not cross-emulate architectures. The manual `arm64_tcg_benchmark` mode is the exception: it boots the same pinned ARM64 Ubuntu image with TCG on x86_64 and ARM64 Linux hosts.
 
 ## Trigger paths
 
@@ -25,13 +28,18 @@ The workflow has two triggers:
 ```yaml
 on:
   workflow_dispatch:
+    inputs:
+      arm64_tcg_benchmark:
+        type: boolean
+        default: false
   workflow_run:
     workflows: ["Release Dev"]
     types: [completed]
 ```
 
 - **Automatic.** Every successful `Release Dev` run (on `main` or a manual dispatch of Release Dev) fires the canary. Each job gates on `github.event.workflow_run.conclusion == 'success'` so a failed Release Dev does not run the canary.
-- **Manual.** `workflow_dispatch` lets you run the canary on demand against any branch's workflow definition.
+- **Manual.** `workflow_dispatch` lets you run the canary on demand against any branch's workflow definition. The Snap job is skipped because a manual run has no triggering Release Dev artifact.
+- **ARM64 TCG benchmark.** A manual dispatch with `arm64_tcg_benchmark=true` skips the artifact canaries and boots the pinned ARM64 Ubuntu guest on both public Linux host architectures. Use it to compare QEMU/TCG startup without involving macOS.
 
 When dispatched manually, `github.event.workflow_run.head_sha` is empty and the workflow falls back to `github.sha` (the branch tip) for the `install.sh` URL.
 
@@ -43,6 +51,14 @@ Run the canary as-is on the current branch:
 gh workflow run release-canary.yml --ref "$(git branch --show-current)"
 ```
 
+Run only the ARM64 TCG boot benchmark:
+
+```shell
+gh workflow run release-canary.yml \
+  --ref "$(git branch --show-current)" \
+  -f arm64_tcg_benchmark=true
+```
+
 Watch the run that starts:
 
 ```shell
@@ -59,9 +75,36 @@ gh run view <run-id> --log-failed
 
 ## Iterating on the canary itself
 
-When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against *main's published artifacts* (`0.0.0-dev` chart, `:dev` images, latest tagged install.sh assets). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts.
+When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against the floating published artifacts (`dev` Linux packages, `0.0.0-dev` chart, and `:dev` images). The macOS job continues to exercise the latest tagged Homebrew release.
+
+Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch are exercised. The selected binary release follows the per-job behavior described above.
+
+## Local Linux VM reproduction
+
+Reproduce each native-architecture Linux canary with the same VM app:
+
+```shell
+nix run .#test-vm -- \
+  --distro ubuntu \
+  --with docker \
+  -- env \
+    OPENSHELL_CANARY_DRIVER=docker \
+    OPENSHELL_CANARY_INSTALL_SH_URL="https://raw.githubusercontent.com/NVIDIA/OpenShell/$(git rev-parse HEAD)/install.sh" \
+  bash -lc '
+    set -euo pipefail
+    mkdir -p "${HOME}/.config/openshell"
+    printf "OPENSHELL_DRIVERS=%s\n" "${OPENSHELL_CANARY_DRIVER}" \
+      > "${HOME}/.config/openshell/gateway.env"
+    curl -LsSf "${OPENSHELL_CANARY_INSTALL_SH_URL}" |
+      OPENSHELL_VERSION=dev sh
+    openshell --version
+    openshell status
+  '
+```
 
-Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch *are* exercised even though the binaries it downloads are from the latest public tag.
+For the RPM path, replace `--distro ubuntu --with docker` with
+`--distro fedora --with podman --with selinux` and set
+`OPENSHELL_CANARY_DRIVER=podman`.
 
 ## Testing artifacts from a specific SHA
 
@@ -107,8 +150,9 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i
 
 | Symptom | Likely cause | Where to look |
 |---|---|---|
-| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Latest tagged release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. |
-| `macos`/`ubuntu`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_DRIVERS` env in the "Ensure …" step. |
+| `macos`/`linux-vm` job fails on `install.sh` | Published release asset missing, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. |
+| `linux-vm` fails before SSH | Cloud-image download, QEMU startup, or guest boot failure. | The runner prints its QEMU and serial logs automatically on failure; check whether it selected KVM or TCG. |
+| `macos`/`linux-vm` job fails on `openshell status` | Local gateway service did not start (systemd/brew/Docker/Podman). Often a driver issue. | Installer diagnostics and the configured `OPENSHELL_DRIVERS` value in the job log. |
 | `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. |
 | `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check `:dev` image existence at `ghcr.io/nvidia/openshell/gateway`. |
 | `kubernetes` job fails on `openshell gateway add` or `status` | Port-forward not reachable, or CLI/gateway proto mismatch. | `port-forward.log` and `openshell gateway list` in the diagnostics dump. |

.github/workflows/release-canary.yml

@@ -2,6 +2,12 @@ name: Release Canary
 
 on:
   workflow_dispatch:
+    inputs:
+      arm64_tcg_benchmark:
+        description: Boot the same ARM64 Ubuntu guest with TCG on x86_64 and ARM64 Linux hosts
+        required: false
+        default: false
+        type: boolean
   workflow_run:
     workflows: ["Release Dev"]
     types: [completed]
@@ -17,7 +23,7 @@ defaults:
 jobs:
   macos:
     name: macOS Homebrew
-    if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
+    if: ${{ (github.event_name == 'workflow_dispatch' && !inputs.arm64_tcg_benchmark) || github.event.workflow_run.conclusion == 'success' }}
     runs-on: macos-latest-xlarge
     timeout-minutes: 20
     steps:
@@ -30,120 +36,119 @@ jobs:
           curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh
           openshell status
 
-  ubuntu:
-    name: Ubuntu Docker
-    if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
-    runs-on: ubuntu-latest
-    timeout-minutes: 20
+  linux-vm:
+    name: Linux VM (${{ matrix.distro }}, ${{ matrix.architecture }})
+    if: ${{ (github.event_name == 'workflow_dispatch' && !inputs.arm64_tcg_benchmark) || github.event.workflow_run.conclusion == 'success' }}
+    runs-on: ${{ matrix.runner }}
+    timeout-minutes: 90
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - runner: ubuntu-24.04
+            architecture: amd64
+            distro: ubuntu
+            driver: docker
+            configurations: docker
+          - runner: ubuntu-24.04-arm
+            architecture: arm64
+            distro: ubuntu
+            driver: docker
+            configurations: docker
+          - runner: ubuntu-24.04
+            architecture: amd64
+            distro: fedora
+            driver: podman
+            configurations: podman selinux
+          - runner: ubuntu-24.04-arm
+            architecture: arm64
+            distro: fedora
+            driver: podman
+            configurations: podman selinux
     steps:
-      - name: Ensure Docker
-        run: |
-          if ! command -v docker >/dev/null 2>&1; then
-            sudo apt-get update
-            sudo apt-get install -y docker.io
-          fi
-          sudo systemctl start docker || sudo service docker start
-          mkdir -p "${HOME}/.config/openshell"
-          printf 'OPENSHELL_DRIVERS=docker\n' > "${HOME}/.config/openshell/gateway.env"
-          docker info
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ github.event.workflow_run.head_sha || github.sha }}
+          persist-credentials: false
 
-      - name: Install and check status
-        run: |
-          curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh
-          openshell status
+      - name: Install Nix
+        uses: cachix/install-nix-action@630ae543ea3a38a9a4166f03376c02c50f408342 # v31.11.0
+        with:
+          enable_kvm: true
+          github_access_token: ${{ github.token }}
 
-  fedora:
-    name: Fedora RPM
-    if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
-    runs-on: linux-amd64-cpu8
-    timeout-minutes: 20
-    env:
-      FEDORA_CANARY_CONTAINER: openshell-fedora-canary-${{ github.run_id }}-${{ github.run_attempt }}
-    steps:
-      - name: Start Fedora systemd container and root user manager
+      - name: Install dev release and check status
+        env:
+          TEST_VM_CONFIGURATIONS: ${{ matrix.configurations }}
+          TEST_VM_DISTRO: ${{ matrix.distro }}
+          TEST_VM_DRIVER: ${{ matrix.driver }}
+          TEST_VM_INSTALL_SH_URL: https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh
         run: |
           set -euo pipefail
 
-          docker run --detach \
-            --name "${FEDORA_CANARY_CONTAINER}" \
-            --privileged \
-            --cgroupns=host \
-            --tmpfs /run \
-            --tmpfs /tmp \
-            --volume /sys/fs/cgroup:/sys/fs/cgroup:rw \
-            fedora:latest \
-            bash -lc 'dnf install -y curl dbus-daemon podman systemd && exec /usr/sbin/init'
-
-          for _ in $(seq 1 120); do
-            if docker exec "${FEDORA_CANARY_CONTAINER}" systemctl list-units --no-pager >/dev/null 2>&1; then
-              break
-            fi
-            if [ "$(docker inspect -f '{{.State.Running}}' "${FEDORA_CANARY_CONTAINER}")" != "true" ]; then
-              echo "::error::Fedora systemd container exited before systemd became reachable"
-              docker logs "${FEDORA_CANARY_CONTAINER}" >&2 || true
-              exit 1
-            fi
-            sleep 1
+          read -r -a configurations <<< "${TEST_VM_CONFIGURATIONS}"
+          configuration_args=()
+          for configuration in "${configurations[@]}"; do
+            configuration_args+=(--with "${configuration}")
           done
 
-          if ! docker exec "${FEDORA_CANARY_CONTAINER}" systemctl list-units --no-pager >/dev/null 2>&1; then
-            echo "::error::Fedora systemd container did not become reachable within 120s"
-            docker logs "${FEDORA_CANARY_CONTAINER}" >&2 || true
-            exit 1
-          fi
+          # The variables in the single-quoted script expand inside the guest.
+          # shellcheck disable=SC2016
+          nix run .#test-vm -- \
+            --distro "${TEST_VM_DISTRO}" \
+            "${configuration_args[@]}" \
+            -- \
+            env \
+              "OPENSHELL_CANARY_DRIVER=${TEST_VM_DRIVER}" \
+              "OPENSHELL_CANARY_INSTALL_SH_URL=${TEST_VM_INSTALL_SH_URL}" \
+            bash -lc '
+              set -euo pipefail
+              mkdir -p "${HOME}/.config/openshell"
+              printf "OPENSHELL_DRIVERS=%s\n" "${OPENSHELL_CANARY_DRIVER}" \
+                > "${HOME}/.config/openshell/gateway.env"
+              curl -LsSf "${OPENSHELL_CANARY_INSTALL_SH_URL}" |
+                OPENSHELL_VERSION=dev sh
+              openshell --version
+              openshell status
+            '
 
-          docker exec --interactive "${FEDORA_CANARY_CONTAINER}" env \
-            HOME=/root \
-            XDG_RUNTIME_DIR=/run/user/0 \
-            DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus \
-            bash -s <<'EOF'
-          set -euo pipefail
-          # install.sh manages the RPM gateway as a systemd user unit. This
-          # container is booted with systemd as PID 1, but it still has no
-          # login session. Start root's user manager explicitly so the
-          # installer can test service restart and gateway registration
-          # instead of its "restart later" fallback.
-          mkdir -p "${XDG_RUNTIME_DIR}"
-          chmod 700 "${XDG_RUNTIME_DIR}"
-          systemctl start user-runtime-dir@0.service || true
-          systemctl start user@0.service
+  arm64-tcg-benchmark:
+    name: ARM64 TCG boot (${{ matrix.host_architecture }} host)
+    if: ${{ github.event_name == 'workflow_dispatch' && inputs.arm64_tcg_benchmark }}
+    runs-on: ${{ matrix.runner }}
+    timeout-minutes: 25
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - runner: ubuntu-24.04
+            host_architecture: x86_64
+          - runner: ubuntu-24.04-arm
+            host_architecture: aarch64
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ github.sha }}
+          persist-credentials: false
 
-          for _ in $(seq 1 30); do
-            if systemctl --user daemon-reload; then
-              break
-            fi
-            sleep 1
-          done
-          if ! systemctl --user daemon-reload; then
-            systemctl status user@0.service --no-pager >&2 || true
-            journalctl -u user@0.service --no-pager -n 80 >&2 || true
-            systemctl --user status --no-pager >&2 || true
-            exit 1
-          fi
-          EOF
+      - name: Install Nix
+        uses: cachix/install-nix-action@630ae543ea3a38a9a4166f03376c02c50f408342 # v31.11.0
+        with:
+          enable_kvm: false
+          github_access_token: ${{ github.token }}
 
-      - name: Install and check status
+      - name: Report host
         run: |
           set -euo pipefail
+          uname -a
+          lscpu
 
-          docker exec --interactive "${FEDORA_CANARY_CONTAINER}" env \
-            HOME=/root \
-            XDG_RUNTIME_DIR=/run/user/0 \
-            DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus \
-            INSTALL_SH_URL="https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh" \
-            bash -s <<'EOF'
-          set -euo pipefail
-          mkdir -p "${HOME}/.config/openshell"
-          printf 'OPENSHELL_DRIVERS=podman\n' > "${HOME}/.config/openshell/gateway.env"
-          podman info
-          curl -LsSf "${INSTALL_SH_URL}" | sh
-          openshell status
-          EOF
-
-      - name: Stop Fedora systemd container
-        if: always()
+      - name: Boot pinned ARM64 Ubuntu guest with TCG
         run: |
-          docker rm -f "${FEDORA_CANARY_CONTAINER}" >/dev/null 2>&1 || true
+          set -euo pipefail
+          nix run .#test-vm-arm64-tcg -- \
+            --distro ubuntu \
+            -- true
 
   ubuntu-snap:
     name: Ubuntu Snap
@@ -197,7 +202,7 @@ jobs:
 
   kubernetes:
     name: Kubernetes Helm (kind)
-    if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
+    if: ${{ (github.event_name == 'workflow_dispatch' && !inputs.arm64_tcg_benchmark) || github.event.workflow_run.conclusion == 'success' }}
     runs-on: ubuntu-latest
     timeout-minutes: 20
     env:

architecture/build.md

@@ -161,6 +161,19 @@ The high-level CI model:
 5. Gate jobs verify that the mirror branch matches the PR head, or that the merge-group workflow ran for the queued SHA, and that the expected non-gate workflow actually ran.
 6. Release workflows rebuild and publish binaries, wheels, images, and docs.
 
+After each successful Release Dev run, the release canary smoke-tests published
+artifacts. Linux package installs run in native-architecture Ubuntu and Fedora
+QEMU guests on public AMD64 and ARM64 GitHub runners. The VM harness uses KVM
+when available and falls back to TCG otherwise. The ARM64 fallback uses QEMU's
+bundled EDK2 firmware, which is validated under TCG on both public Linux host
+architectures. Separate jobs retain macOS Homebrew, Ubuntu Snap, and kind-based
+Helm coverage.
+
+A manual `arm64_tcg_benchmark` dispatch mode skips the artifact canaries and
+boots the same pinned ARM64 Ubuntu cloud image under QEMU TCG on public x86_64
+and ARM64 Linux runners. The comparison keeps the native ARM64 runner as the
+production host while verifying that x86_64 TCG remains a viable fallback.
+
 See `CI.md` for the contributor workflow, labels, and maintainer merge-queue workflow.
 
 ## Docs Site

flake.nix

@@ -38,9 +38,20 @@
           programs.nixfmt.enable = true;
         };
         testVm = import ./nix/vm { inherit pkgs; };
+        testVmArm64Tcg = import ./nix/vm {
+          inherit pkgs;
+          accelerator = "tcg";
+          architecture = "aarch64";
+          useQemuFirmware = true;
+        };
       in
       {
-        apps.test-vm = testVm.app;
+        apps = {
+          test-vm = testVm.app;
+        }
+        // pkgs.lib.optionalAttrs pkgs.stdenv.hostPlatform.isLinux {
+          test-vm-arm64-tcg = testVmArm64Tcg.app;
+        };
 
         devShells.default = pkgs.mkShell {
           packages = with pkgs; [