Skip to content

[DECOUPLED-MODE] ci: run the decoupled test gate in a real decoupled environment - #4658

Open
gulsumgudukbay wants to merge 7 commits into
AI-Hypercomputer:mainfrom
ROCm:decoupled-mode-testing
Open

[DECOUPLED-MODE] ci: run the decoupled test gate in a real decoupled environment#4658
gulsumgudukbay wants to merge 7 commits into
AI-Hypercomputer:mainfrom
ROCm:decoupled-mode-testing

Conversation

@gulsumgudukbay

@gulsumgudukbay gulsumgudukbay commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Continues #4642, which I was asked to take over. I don't have push access to that branch (it lives in this repo, not a fork), so this comes from the ROCm fork and carries @shralex's commits plus one on top addressing the review comments.

Summary

  • The gate now builds its own venv from the new generated_requirements/decoupled-requirements.txt and fails the step if
    google.cloud.storage, google.api_core, google.auth, gcsfs, jetstream or the diagnostics clients are importable in it. DECOUPLE_GCLOUD=TRUE only changes what gcloud_stub hands back, so a gate running in a venv that has google-cloud-storage installed would have passed Checkpoint_conversion: import GCS via gcloud_stub for decoupled mode #4003, [DECOUPLED-MODE] grpo_trainer: route goodput/vertex imports through gcloud_stub for decoupled #4180 and [DECOUPLED-MODE] checkpoint_conversion: import GCS via gcloud_stub in load_dynamic #4637.
  • decoupled-requirements.txt is derived from the GPU pre-training lock minus the Google Cloud clients and the accelerator wheels, so the decoupled run uses the same versions as the gpu-unit job (checked against a gpu-unit run on main, no version differences) and installs on any hardware. A decoupled run on an accelerator adds the dropped entries back via --print-accelerator-requirements. Deriving it from a lock rather than from base requirements keeps regeneration offline, which is what lets the new decoupled-requirements pre-commit hook fail any requirements change that leaves it stale. Replaces requirements_decoupled_jax_0_7.1.txt, which nothing referenced.
  • Step wiring fixes: it sat between the previous step's run: and its env:, so Run Tests lost all ten INPUTS_* variables; it ran in all four cpu-unit shards; and $PYTHON_EXE was empty across the step boundary.
  • Drops decoupled_target from the integration test and from the files pytest.ini ignores, where the marker never took effect, and corrects the marker docs.

Prerequisite

#4637 has to land first. Without it, tests/unit/checkpointing_test.py cannot be collected once the Google Cloud packages are genuinely absent, and the whole runxits 2.

Test plan

  • Ran the step's run: block as is under bash -e against a built wheel, with [DECOUPLED-MODE] checkpoint_conversion: import GCS via gcloud_stub in load_dynamic #4637 applied: 46 passed, 2 skipped, no collection errors.
  • Guard negative cases: with google-cloud-storage present it exits 1 and stops the step before pytest; same for gcsfs; a missing wheel fails loudly rather than skipping.
  • The same selection with DECOUPLE_GCLOUD unset fails at collection (ModuleNotFoundError: ml_goodput_measurement), so the gate can't pass trivially.
  • pre-commit run passes on every touched file, including the new hook.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@gulsumgudukbay gulsumgudukbay changed the title ci: run the decoupled test gate in a real decoupled environment [DECOUPLED-MODE] ci: run the decoupled test gate in a real decoupled environment Jul 29, 2026
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

shralex added 4 commits July 29, 2026 17:31
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI.
- Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target].
- Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI.
- Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target].
- Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI.
- Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target].
- Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI.
- Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target].
- Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
shralex and others added 3 commits July 29, 2026 17:31
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI.
- Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target].
- Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI.
- Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target].
- Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
DECOUPLE_GCLOUD=TRUE only changes what gcloud_stub returns, so running the gate in a
venv that has google-cloud-storage installed let module-scope Google Cloud imports
resolve; AI-Hypercomputer#4003, AI-Hypercomputer#4180 and AI-Hypercomputer#4637 would all have passed it. The step now builds its own
venv from generated_requirements/decoupled-requirements.txt - the GPU lock minus the
Google Cloud packages and the accelerator entries, kept current by a pre-commit hook -
and fails if google.cloud.storage, google.api_core, google.auth, gcsfs, jetstream or
the diagnostics clients are importable in it. Replaces the unreferenced jax 0.7.1
decoupled requirements file.

Also fixes the wiring: the step sat between the previous step's run: and its env:, so
Run Tests lost all ten INPUTS_* variables; it ran in all four cpu-unit shards; and
$PYTHON_EXE was empty across the step boundary.

Drops decoupled_target from the integration test and the four files pytest.ini
ignores, where it never took effect, and corrects the marker docs. AI-Hypercomputer#4637 is a
prerequisite; with it applied the suite is 46 passed, 2 skipped.
@gulsumgudukbay
gulsumgudukbay force-pushed the decoupled-mode-testing branch from 3cf898a to a4ff38e Compare July 29, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants