diff --git a/.agents/skills/test-release-canary/SKILL.md b/.agents/skills/test-release-canary/SKILL.md index 4bf7d38ae3..03fff93892 100644 --- a/.agents/skills/test-release-canary/SKILL.md +++ b/.agents/skills/test-release-canary/SKILL.md @@ -12,11 +12,14 @@ The Release Canary (`.github/workflows/release-canary.yml`) smoke-tests the arti | Job | Runner | Verifies | |---|---|---| | `macos` | `macos-latest-xlarge` | `install.sh` resolves the Homebrew formula, brew installs the cask, and `openshell status` reaches the brew-services–backed local gateway with the VM driver. | -| `ubuntu` | `ubuntu-latest` | `install.sh` installs the Debian package, the post-install systemd user service starts, and `openshell status` reaches the local gateway with the Docker driver. | -| `fedora` | `fedora:latest` container | `install.sh` installs the RPM packages, the local gateway starts under Podman, and `openshell status` succeeds. | +| `linux-vm` (Ubuntu) | `ubuntu-24.04` / `ubuntu-24.04-arm` | The Nix VM harness boots Ubuntu 24.04 on AMD64 and ARM64, configures Docker, installs the rolling `dev` Debian packages, and verifies the local gateway. | +| `linux-vm` (Fedora) | `ubuntu-24.04` / `ubuntu-24.04-arm` | The Nix VM harness boots Fedora 44 on AMD64 and ARM64, configures rootless Podman with enforcing SELinux, installs the rolling `dev` RPM packages, and verifies the local gateway. | +| `ubuntu-snap` | `ubuntu-latest` | The Snap produced by the triggering Release Dev run installs with its required interfaces and reaches the Docker-backed gateway. | | `kubernetes` | `ubuntu-latest` + kind | `helm install oci://ghcr.io/nvidia/openshell/helm-chart --version 0.0.0-dev` succeeds in a kind cluster, the gateway pod becomes Ready, port-forward exposes 8080, and the released CLI registers the in-cluster gateway and runs `openshell status` against it. | -`install.sh` defaults to the *latest tagged* release — the canary is therefore checking that the most recent public release still installs, not the just-published `dev` build. The `kubernetes` job is the exception: it pins to `0.0.0-dev` chart + `:dev` images. +The Linux VM matrix sets `OPENSHELL_VERSION=dev`, so it tests the packages published by Release Dev. The Snap job downloads artifacts from the triggering run, and Kubernetes pins the `0.0.0-dev` chart and `:dev` images. The macOS job retains the installer's default latest-tagged release behavior. + +On Linux, the VM harness uses KVM when `/dev/kvm` is readable and writable. It falls back automatically to native-architecture QEMU TCG when KVM is unavailable, which is expected on public ARM64 runners. The ARM64 fallback switches from the default firmware to QEMU's bundled EDK2 firmware, which is validated under TCG. Normal canaries do not cross-emulate architectures. The manual `arm64_tcg_benchmark` mode is the exception: it boots the same pinned ARM64 Ubuntu image with TCG on x86_64 and ARM64 Linux hosts. ## Trigger paths @@ -25,13 +28,18 @@ The workflow has two triggers: ```yaml on: workflow_dispatch: + inputs: + arm64_tcg_benchmark: + type: boolean + default: false workflow_run: workflows: ["Release Dev"] types: [completed] ``` - **Automatic.** Every successful `Release Dev` run (on `main` or a manual dispatch of Release Dev) fires the canary. Each job gates on `github.event.workflow_run.conclusion == 'success'` so a failed Release Dev does not run the canary. -- **Manual.** `workflow_dispatch` lets you run the canary on demand against any branch's workflow definition. +- **Manual.** `workflow_dispatch` lets you run the canary on demand against any branch's workflow definition. The Snap job is skipped because a manual run has no triggering Release Dev artifact. +- **ARM64 TCG benchmark.** A manual dispatch with `arm64_tcg_benchmark=true` skips the artifact canaries and boots the pinned ARM64 Ubuntu guest on both public Linux host architectures. Use it to compare QEMU/TCG startup without involving macOS. When dispatched manually, `github.event.workflow_run.head_sha` is empty and the workflow falls back to `github.sha` (the branch tip) for the `install.sh` URL. @@ -43,6 +51,14 @@ Run the canary as-is on the current branch: gh workflow run release-canary.yml --ref "$(git branch --show-current)" ``` +Run only the ARM64 TCG boot benchmark: + +```shell +gh workflow run release-canary.yml \ + --ref "$(git branch --show-current)" \ + -f arm64_tcg_benchmark=true +``` + Watch the run that starts: ```shell @@ -59,9 +75,36 @@ gh run view --log-failed ## Iterating on the canary itself -When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against *main's published artifacts* (`0.0.0-dev` chart, `:dev` images, latest tagged install.sh assets). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts. +When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against the floating published artifacts (`dev` Linux packages, `0.0.0-dev` chart, and `:dev` images). The macOS job continues to exercise the latest tagged Homebrew release. + +Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch are exercised. The selected binary release follows the per-job behavior described above. + +## Local Linux VM reproduction + +Reproduce each native-architecture Linux canary with the same VM app: + +```shell +nix run .#test-vm -- \ + --distro ubuntu \ + --with docker \ + -- env \ + OPENSHELL_CANARY_DRIVER=docker \ + OPENSHELL_CANARY_INSTALL_SH_URL="https://raw.githubusercontent.com/NVIDIA/OpenShell/$(git rev-parse HEAD)/install.sh" \ + bash -lc ' + set -euo pipefail + mkdir -p "${HOME}/.config/openshell" + printf "OPENSHELL_DRIVERS=%s\n" "${OPENSHELL_CANARY_DRIVER}" \ + > "${HOME}/.config/openshell/gateway.env" + curl -LsSf "${OPENSHELL_CANARY_INSTALL_SH_URL}" | + OPENSHELL_VERSION=dev sh + openshell --version + openshell status + ' +``` -Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch *are* exercised even though the binaries it downloads are from the latest public tag. +For the RPM path, replace `--distro ubuntu --with docker` with +`--distro fedora --with podman --with selinux` and set +`OPENSHELL_CANARY_DRIVER=podman`. ## Testing artifacts from a specific SHA @@ -107,8 +150,9 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i | Symptom | Likely cause | Where to look | |---|---|---| -| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Latest tagged release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. | -| `macos`/`ubuntu`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_DRIVERS` env in the "Ensure …" step. | +| `macos`/`linux-vm` job fails on `install.sh` | Published release asset missing, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. | +| `linux-vm` fails before SSH | Cloud-image download, QEMU startup, or guest boot failure. | The runner prints its QEMU and serial logs automatically on failure; check whether it selected KVM or TCG. | +| `macos`/`linux-vm` job fails on `openshell status` | Local gateway service did not start (systemd/brew/Docker/Podman). Often a driver issue. | Installer diagnostics and the configured `OPENSHELL_DRIVERS` value in the job log. | | `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. | | `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check `:dev` image existence at `ghcr.io/nvidia/openshell/gateway`. | | `kubernetes` job fails on `openshell gateway add` or `status` | Port-forward not reachable, or CLI/gateway proto mismatch. | `port-forward.log` and `openshell gateway list` in the diagnostics dump. | diff --git a/.github/workflows/release-canary.yml b/.github/workflows/release-canary.yml index 896d12d190..ad8dc6795e 100644 --- a/.github/workflows/release-canary.yml +++ b/.github/workflows/release-canary.yml @@ -2,6 +2,12 @@ name: Release Canary on: workflow_dispatch: + inputs: + arm64_tcg_benchmark: + description: Boot the same ARM64 Ubuntu guest with TCG on x86_64 and ARM64 Linux hosts + required: false + default: false + type: boolean workflow_run: workflows: ["Release Dev"] types: [completed] @@ -17,7 +23,7 @@ defaults: jobs: macos: name: macOS Homebrew - if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} + if: ${{ (github.event_name == 'workflow_dispatch' && !inputs.arm64_tcg_benchmark) || github.event.workflow_run.conclusion == 'success' }} runs-on: macos-latest-xlarge timeout-minutes: 20 steps: @@ -30,120 +36,119 @@ jobs: curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh openshell status - ubuntu: - name: Ubuntu Docker - if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} - runs-on: ubuntu-latest - timeout-minutes: 20 + linux-vm: + name: Linux VM (${{ matrix.distro }}, ${{ matrix.architecture }}) + if: ${{ (github.event_name == 'workflow_dispatch' && !inputs.arm64_tcg_benchmark) || github.event.workflow_run.conclusion == 'success' }} + runs-on: ${{ matrix.runner }} + timeout-minutes: 90 + strategy: + fail-fast: false + matrix: + include: + - runner: ubuntu-24.04 + architecture: amd64 + distro: ubuntu + driver: docker + configurations: docker + - runner: ubuntu-24.04-arm + architecture: arm64 + distro: ubuntu + driver: docker + configurations: docker + - runner: ubuntu-24.04 + architecture: amd64 + distro: fedora + driver: podman + configurations: podman selinux + - runner: ubuntu-24.04-arm + architecture: arm64 + distro: fedora + driver: podman + configurations: podman selinux steps: - - name: Ensure Docker - run: | - if ! command -v docker >/dev/null 2>&1; then - sudo apt-get update - sudo apt-get install -y docker.io - fi - sudo systemctl start docker || sudo service docker start - mkdir -p "${HOME}/.config/openshell" - printf 'OPENSHELL_DRIVERS=docker\n' > "${HOME}/.config/openshell/gateway.env" - docker info + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.event.workflow_run.head_sha || github.sha }} + persist-credentials: false - - name: Install and check status - run: | - curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh - openshell status + - name: Install Nix + uses: cachix/install-nix-action@630ae543ea3a38a9a4166f03376c02c50f408342 # v31.11.0 + with: + enable_kvm: true + github_access_token: ${{ github.token }} - fedora: - name: Fedora RPM - if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} - runs-on: linux-amd64-cpu8 - timeout-minutes: 20 - env: - FEDORA_CANARY_CONTAINER: openshell-fedora-canary-${{ github.run_id }}-${{ github.run_attempt }} - steps: - - name: Start Fedora systemd container and root user manager + - name: Install dev release and check status + env: + TEST_VM_CONFIGURATIONS: ${{ matrix.configurations }} + TEST_VM_DISTRO: ${{ matrix.distro }} + TEST_VM_DRIVER: ${{ matrix.driver }} + TEST_VM_INSTALL_SH_URL: https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh run: | set -euo pipefail - docker run --detach \ - --name "${FEDORA_CANARY_CONTAINER}" \ - --privileged \ - --cgroupns=host \ - --tmpfs /run \ - --tmpfs /tmp \ - --volume /sys/fs/cgroup:/sys/fs/cgroup:rw \ - fedora:latest \ - bash -lc 'dnf install -y curl dbus-daemon podman systemd && exec /usr/sbin/init' - - for _ in $(seq 1 120); do - if docker exec "${FEDORA_CANARY_CONTAINER}" systemctl list-units --no-pager >/dev/null 2>&1; then - break - fi - if [ "$(docker inspect -f '{{.State.Running}}' "${FEDORA_CANARY_CONTAINER}")" != "true" ]; then - echo "::error::Fedora systemd container exited before systemd became reachable" - docker logs "${FEDORA_CANARY_CONTAINER}" >&2 || true - exit 1 - fi - sleep 1 + read -r -a configurations <<< "${TEST_VM_CONFIGURATIONS}" + configuration_args=() + for configuration in "${configurations[@]}"; do + configuration_args+=(--with "${configuration}") done - if ! docker exec "${FEDORA_CANARY_CONTAINER}" systemctl list-units --no-pager >/dev/null 2>&1; then - echo "::error::Fedora systemd container did not become reachable within 120s" - docker logs "${FEDORA_CANARY_CONTAINER}" >&2 || true - exit 1 - fi + # The variables in the single-quoted script expand inside the guest. + # shellcheck disable=SC2016 + nix run .#test-vm -- \ + --distro "${TEST_VM_DISTRO}" \ + "${configuration_args[@]}" \ + -- \ + env \ + "OPENSHELL_CANARY_DRIVER=${TEST_VM_DRIVER}" \ + "OPENSHELL_CANARY_INSTALL_SH_URL=${TEST_VM_INSTALL_SH_URL}" \ + bash -lc ' + set -euo pipefail + mkdir -p "${HOME}/.config/openshell" + printf "OPENSHELL_DRIVERS=%s\n" "${OPENSHELL_CANARY_DRIVER}" \ + > "${HOME}/.config/openshell/gateway.env" + curl -LsSf "${OPENSHELL_CANARY_INSTALL_SH_URL}" | + OPENSHELL_VERSION=dev sh + openshell --version + openshell status + ' - docker exec --interactive "${FEDORA_CANARY_CONTAINER}" env \ - HOME=/root \ - XDG_RUNTIME_DIR=/run/user/0 \ - DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus \ - bash -s <<'EOF' - set -euo pipefail - # install.sh manages the RPM gateway as a systemd user unit. This - # container is booted with systemd as PID 1, but it still has no - # login session. Start root's user manager explicitly so the - # installer can test service restart and gateway registration - # instead of its "restart later" fallback. - mkdir -p "${XDG_RUNTIME_DIR}" - chmod 700 "${XDG_RUNTIME_DIR}" - systemctl start user-runtime-dir@0.service || true - systemctl start user@0.service + arm64-tcg-benchmark: + name: ARM64 TCG boot (${{ matrix.host_architecture }} host) + if: ${{ github.event_name == 'workflow_dispatch' && inputs.arm64_tcg_benchmark }} + runs-on: ${{ matrix.runner }} + timeout-minutes: 25 + strategy: + fail-fast: false + matrix: + include: + - runner: ubuntu-24.04 + host_architecture: x86_64 + - runner: ubuntu-24.04-arm + host_architecture: aarch64 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.sha }} + persist-credentials: false - for _ in $(seq 1 30); do - if systemctl --user daemon-reload; then - break - fi - sleep 1 - done - if ! systemctl --user daemon-reload; then - systemctl status user@0.service --no-pager >&2 || true - journalctl -u user@0.service --no-pager -n 80 >&2 || true - systemctl --user status --no-pager >&2 || true - exit 1 - fi - EOF + - name: Install Nix + uses: cachix/install-nix-action@630ae543ea3a38a9a4166f03376c02c50f408342 # v31.11.0 + with: + enable_kvm: false + github_access_token: ${{ github.token }} - - name: Install and check status + - name: Report host run: | set -euo pipefail + uname -a + lscpu - docker exec --interactive "${FEDORA_CANARY_CONTAINER}" env \ - HOME=/root \ - XDG_RUNTIME_DIR=/run/user/0 \ - DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus \ - INSTALL_SH_URL="https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh" \ - bash -s <<'EOF' - set -euo pipefail - mkdir -p "${HOME}/.config/openshell" - printf 'OPENSHELL_DRIVERS=podman\n' > "${HOME}/.config/openshell/gateway.env" - podman info - curl -LsSf "${INSTALL_SH_URL}" | sh - openshell status - EOF - - - name: Stop Fedora systemd container - if: always() + - name: Boot pinned ARM64 Ubuntu guest with TCG run: | - docker rm -f "${FEDORA_CANARY_CONTAINER}" >/dev/null 2>&1 || true + set -euo pipefail + nix run .#test-vm-arm64-tcg -- \ + --distro ubuntu \ + -- true ubuntu-snap: name: Ubuntu Snap @@ -197,7 +202,7 @@ jobs: kubernetes: name: Kubernetes Helm (kind) - if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} + if: ${{ (github.event_name == 'workflow_dispatch' && !inputs.arm64_tcg_benchmark) || github.event.workflow_run.conclusion == 'success' }} runs-on: ubuntu-latest timeout-minutes: 20 env: diff --git a/architecture/build.md b/architecture/build.md index eb8672f15a..fd545b8597 100644 --- a/architecture/build.md +++ b/architecture/build.md @@ -161,6 +161,19 @@ The high-level CI model: 5. Gate jobs verify that the mirror branch matches the PR head, or that the merge-group workflow ran for the queued SHA, and that the expected non-gate workflow actually ran. 6. Release workflows rebuild and publish binaries, wheels, images, and docs. +After each successful Release Dev run, the release canary smoke-tests published +artifacts. Linux package installs run in native-architecture Ubuntu and Fedora +QEMU guests on public AMD64 and ARM64 GitHub runners. The VM harness uses KVM +when available and falls back to TCG otherwise. The ARM64 fallback uses QEMU's +bundled EDK2 firmware, which is validated under TCG on both public Linux host +architectures. Separate jobs retain macOS Homebrew, Ubuntu Snap, and kind-based +Helm coverage. + +A manual `arm64_tcg_benchmark` dispatch mode skips the artifact canaries and +boots the same pinned ARM64 Ubuntu cloud image under QEMU TCG on public x86_64 +and ARM64 Linux runners. The comparison keeps the native ARM64 runner as the +production host while verifying that x86_64 TCG remains a viable fallback. + See `CI.md` for the contributor workflow, labels, and maintainer merge-queue workflow. ## Docs Site diff --git a/flake.nix b/flake.nix index aa4ce8829a..1ad57abfe5 100644 --- a/flake.nix +++ b/flake.nix @@ -38,9 +38,20 @@ programs.nixfmt.enable = true; }; testVm = import ./nix/vm { inherit pkgs; }; + testVmArm64Tcg = import ./nix/vm { + inherit pkgs; + accelerator = "tcg"; + architecture = "aarch64"; + useQemuFirmware = true; + }; in { - apps.test-vm = testVm.app; + apps = { + test-vm = testVm.app; + } + // pkgs.lib.optionalAttrs pkgs.stdenv.hostPlatform.isLinux { + test-vm-arm64-tcg = testVmArm64Tcg.app; + }; devShells.default = pkgs.mkShell { packages = with pkgs; [ diff --git a/nix/vm/README.md b/nix/vm/README.md index 444c9c5b4f..0ad02028b4 100644 --- a/nix/vm/README.md +++ b/nix/vm/README.md @@ -5,14 +5,14 @@ SPDX-License-Identifier: Apache-2.0 # Test VMs -This prototype uses Nix, QEMU, and Ansible to boot and configure disposable Linux VMs for testing OpenShell packages and binaries. It supports HVF on Apple Silicon macOS, KVM on native-architecture Linux hosts, and a slower TCG fallback on Linux when KVM is unavailable. +This prototype uses Nix, QEMU, and Ansible to boot and configure disposable Linux VMs for testing OpenShell packages and binaries. It supports HVF on Apple Silicon macOS, KVM on native-architecture Linux hosts, and a slower TCG fallback on Linux when KVM is unavailable. The ARM64 fallback switches to QEMU's bundled EDK2 firmware, which is validated with TCG. A separate Linux-only app boots the pinned ARM64 guest with TCG on either an x86_64 or ARM64 host for performance comparisons. ## Requirements - Nix with flakes enabled. -- Apple Silicon macOS with HVF, or a native-architecture Linux host. Linux uses KVM when `/dev/kvm` is available and falls back to QEMU TCG otherwise. +- Apple Silicon macOS with HVF, or a Linux host. The default app uses KVM when `/dev/kvm` is available and falls back to QEMU TCG otherwise. - Enough local capacity for a four-vCPU, 4 GiB guest and a disposable disk overlay. -- Native-architecture artifacts. TCG emulates the guest CPU on Linux but does not enable cross-architecture guests. +- Artifacts matching the guest architecture. The first run downloads the selected cloud image and VM runtime. Nix reuses those immutable inputs on later runs, while each guest starts from a fresh writable overlay. @@ -40,7 +40,7 @@ nix/vm/ - `configuration/*.yml` are host-executed Ansible playbooks that layer optional capabilities onto a base guest. Configurations remain independent and run in the order supplied with repeated `--with` arguments. - `README.md` documents the supported combinations and developer interface. -The root [`flake.nix`](../../flake.nix) exposes this directory as the `test-vm` app. Debian artifact creation remains outside the VM harness in [`tasks/scripts/package-deb.sh`](../../tasks/scripts/package-deb.sh); the runner only installs or copies artifacts that already exist. +The root [`flake.nix`](../../flake.nix) exposes this directory as the native-architecture `test-vm` app and the Linux-only `test-vm-arm64-tcg` app. Debian artifact creation remains outside the VM harness in [`tasks/scripts/package-deb.sh`](../../tasks/scripts/package-deb.sh); the runner only installs or copies artifacts that already exist. ## Supported configurations @@ -57,6 +57,18 @@ List the available distros and configurations: nix run .#test-vm -- --list ``` +## Benchmark ARM64 TCG across Linux hosts + +Boot the pinned ARM64 Ubuntu image with QEMU TCG, validate the guest over SSH, and shut it down: + +```shell +nix run .#test-vm-arm64-tcg -- \ + --distro ubuntu \ + -- true +``` + +Run the same command on x86_64 and ARM64 Linux hosts to compare TCG boot behavior. The app holds the ARM64 cloud image, QEMU version, firmware, vCPU count, and memory constant. The runner prints the elapsed time from QEMU startup until SSH becomes ready. + ## Open an interactive VM Boot a base Ubuntu VM: @@ -129,7 +141,7 @@ nix run .#test-vm -- \ -- openshell --version ``` -For an x86_64 Linux guest, supply x86_64 binaries and use `package:deb:amd64`. The package architecture must match the host and guest architecture. +For an x86_64 Linux guest, supply x86_64 binaries and use `package:deb:amd64`. The package architecture must match the guest architecture. `--install` is repeatable. Debian packages are accepted by Ubuntu; RPM packages are accepted by CentOS, Fedora, and Rocky Linux. This prototype can install an existing RPM but does not build one. @@ -165,7 +177,7 @@ Arguments after `--` are executed inside the guest. Without a command, the runne Nix downloads and caches the selected, hash-pinned cloud image. Each invocation then: 1. Creates a temporary QCOW2 overlay. -2. Boots QEMU with HVF, KVM, or the Linux TCG fallback. +2. Boots QEMU with HVF, KVM, or the Linux TCG fallback. Native ARM64 Linux guests use QEMU's bundled EDK2 firmware when KVM is unavailable. 3. Creates an ephemeral `openshell` user and SSH key through cloud-init. 4. Applies the selected Ansible configurations. 5. Installs or copies the supplied artifacts. @@ -176,7 +188,7 @@ Use `--keep` to preserve the overlay, cloud-init seed, SSH key, and serial log f ## Current limitations -- Host and guest architectures must match. +- The default `test-vm` app uses a guest matching the host architecture. `test-vm-arm64-tcg` is the supported cross-architecture exception on Linux. - TCG is slower than hardware virtualization and uses a longer SSH readiness timeout. - Configurations are applied fresh on every run; prepared VM caching is not implemented. - Only loopback SSH is forwarded to the host; guest gateway ports are not exposed. diff --git a/nix/vm/default.nix b/nix/vm/default.nix index ec82b4b8be..382803bfb9 100644 --- a/nix/vm/default.nix +++ b/nix/vm/default.nix @@ -3,15 +3,37 @@ # PROTOTYPE: Composable distro VMs for installing and exercising artifacts. -{ pkgs }: +{ + pkgs, + architecture ? if pkgs.stdenv.hostPlatform.isAarch64 then "aarch64" else "x86_64", + accelerator ? null, + useQemuFirmware ? false, +}: let - isAarch64 = pkgs.stdenv.hostPlatform.isAarch64; + hostArchitecture = if pkgs.stdenv.hostPlatform.isAarch64 then "aarch64" else "x86_64"; + isAarch64 = architecture == "aarch64"; isDarwin = pkgs.stdenv.hostPlatform.isDarwin; - architecture = if isAarch64 then "aarch64" else "x86_64"; - qemu = pkgs.qemu.override { hostCpuOnly = true; }; + guestMatchesHost = architecture == hostArchitecture; + qemu = pkgs.qemu.override { hostCpuOnly = guestMatchesHost; }; qemuBinary = if isAarch64 then "${qemu}/bin/qemu-system-aarch64" else "${qemu}/bin/qemu-system-x86_64"; + selectedAccelerator = + if accelerator != null then + accelerator + else if isDarwin then + if guestMatchesHost then "hvf" else "tcg" + else if guestMatchesHost then + "kvm" + else + "tcg"; + firmwareCode = + if useQemuFirmware && isAarch64 then + "${qemu}/share/qemu/edk2-aarch64-code.fd" + else + pkgs.OVMF.firmware; + firmwareVars = + if useQemuFirmware && isAarch64 then "${qemu}/share/qemu/edk2-arm-vars.fd" else pkgs.OVMF.variables; distros = { ubuntu = import ./distros/ubuntu.nix { inherit pkgs architecture; }; @@ -64,10 +86,14 @@ let export OPENSHELL_TEST_VM_DISTROS=${distroCatalog} export OPENSHELL_TEST_VM_CONFIGURATIONS=${configurationCatalog} export TEST_VM_QEMU=${qemuBinary} - export TEST_VM_FIRMWARE_CODE=${pkgs.OVMF.firmware} - export TEST_VM_FIRMWARE_VARS=${pkgs.OVMF.variables} + export TEST_VM_FIRMWARE_CODE=${firmwareCode} + export TEST_VM_FIRMWARE_VARS=${firmwareVars} + ${pkgs.lib.optionalString isAarch64 '' + export TEST_VM_TCG_FIRMWARE_CODE=${qemu}/share/qemu/edk2-aarch64-code.fd + export TEST_VM_TCG_FIRMWARE_VARS=${qemu}/share/qemu/edk2-arm-vars.fd + ''} export TEST_VM_MACHINE=${if isAarch64 then "virt" else "q35"} - export TEST_VM_ACCELERATOR=${if isDarwin then "hvf" else "kvm"} + export TEST_VM_ACCELERATOR=${selectedAccelerator} export TEST_VM_ARCHITECTURE=${architecture} exec ${pkgs.bash}/bin/bash ${./run.sh} "$@" ''; diff --git a/nix/vm/run.sh b/nix/vm/run.sh index 1a54ea4a33..f90365366f 100644 --- a/nix/vm/run.sh +++ b/nix/vm/run.sh @@ -166,14 +166,29 @@ for copy_spec in "${copies[@]}"; do esac done -test_vm_cpu=host -ssh_wait_seconds=180 +case "${TEST_VM_ACCELERATOR}" in +tcg) + test_vm_cpu=max + ssh_wait_seconds=600 + ;; +*) + test_vm_cpu=host + ssh_wait_seconds=180 + ;; +esac if [ "${TEST_VM_ACCELERATOR}" = kvm ] && { [ ! -c /dev/kvm ] || [ ! -r /dev/kvm ] || [ ! -w /dev/kvm ]; }; then echo "==> /dev/kvm is unavailable; falling back to QEMU/TCG" TEST_VM_ACCELERATOR=tcg test_vm_cpu=max ssh_wait_seconds=600 + if [ "${TEST_VM_ARCHITECTURE}" = aarch64 ]; then + : "${TEST_VM_TCG_FIRMWARE_CODE:?missing QEMU ARM TCG firmware code}" + : "${TEST_VM_TCG_FIRMWARE_VARS:?missing QEMU ARM TCG firmware variables}" + TEST_VM_FIRMWARE_CODE=${TEST_VM_TCG_FIRMWARE_CODE} + TEST_VM_FIRMWARE_VARS=${TEST_VM_TCG_FIRMWARE_VARS} + echo "==> Using QEMU-bundled ARM firmware for TCG fallback" + fi fi echo "==> Realizing the pinned ${distro} cloud image" @@ -262,6 +277,7 @@ qemu-img resize -q "${overlay}" +16G cp "${TEST_VM_FIRMWARE_VARS}" "${vars}" chmod 0600 "${vars}" +boot_started_seconds=${SECONDS} for attempt in $(seq 1 5); do ssh_port=${requested_ssh_port:-$(pick_free_port)} : >"${qemu_log}" @@ -311,7 +327,7 @@ ssh_args=( -i "${private_key}" -p "${ssh_port}" -o BatchMode=yes - -o ConnectTimeout=5 + -o ConnectTimeout=2 -o LogLevel=ERROR -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null @@ -321,7 +337,7 @@ scp_args=( -i "${private_key}" -P "${ssh_port}" -o BatchMode=yes - -o ConnectTimeout=5 + -o ConnectTimeout=2 -o LogLevel=ERROR -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null @@ -329,7 +345,8 @@ scp_args=( echo "==> Waiting up to ${ssh_wait_seconds} seconds for SSH on 127.0.0.1:${ssh_port}" ssh_ready=0 -for _ in $(seq 1 "$((ssh_wait_seconds / 2))"); do +ssh_deadline_seconds=$((boot_started_seconds + ssh_wait_seconds)) +while [ "${SECONDS}" -lt "${ssh_deadline_seconds}" ]; do if ! kill -0 "${qemu_pid}" 2>/dev/null; then wait "${qemu_pid}" || true qemu_pid= @@ -340,12 +357,18 @@ for _ in $(seq 1 "$((ssh_wait_seconds / 2))"); do ssh_ready=1 break fi - sleep 2 + remaining_seconds=$((ssh_deadline_seconds - SECONDS)) + if [ "${remaining_seconds}" -gt 2 ]; then + sleep 2 + elif [ "${remaining_seconds}" -gt 0 ]; then + sleep "${remaining_seconds}" + fi done if [ "${ssh_ready}" -ne 1 ]; then echo "SSH did not become ready within ${ssh_wait_seconds} seconds" >&2 exit 1 fi +echo "==> SSH ready after $((SECONDS - boot_started_seconds)) seconds" echo "==> Validating ${distro}" # Profile values come from the trusted Nix-generated catalog.