[DECOUPLED-MODE] ci: run the decoupled test gate in a real decoupled environment - #4658
Open
gulsumgudukbay wants to merge 7 commits into
Open
[DECOUPLED-MODE] ci: run the decoupled test gate in a real decoupled environment#4658gulsumgudukbay wants to merge 7 commits into
gulsumgudukbay wants to merge 7 commits into
Conversation
gulsumgudukbay
requested review from
gobbleturk,
khatwanimohit and
shralex
as code owners
July 29, 2026 05:25
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
gulsumgudukbay
requested review from
A9isha,
NuojCheng,
RissyRan,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
bvandermoon,
darisoy,
dipannita08,
gagika,
hengtaoguo,
huytransformer,
igorts-git,
jacoguzo,
jiangjy1982,
parambole,
richjames0,
suexu1025,
vipannalla and
xibinliu
as code owners
July 29, 2026 05:25
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI. - Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target]. - Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI. - Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target]. - Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI. - Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target]. - Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI. - Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target]. - Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI. - Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target]. - Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
- Register decoupled_target marker in pytest.ini for tests to be additionally validated in decoupled mode on CPU in CI. - Mark targeted test modules in unit/ and integration/ with pytestmark = [pytest.mark.decoupled_target]. - Add step in .github/workflows/run_tests_against_package.yml to execute pytest -m decoupled_target -n auto on CPU runners.
DECOUPLE_GCLOUD=TRUE only changes what gcloud_stub returns, so running the gate in a venv that has google-cloud-storage installed let module-scope Google Cloud imports resolve; AI-Hypercomputer#4003, AI-Hypercomputer#4180 and AI-Hypercomputer#4637 would all have passed it. The step now builds its own venv from generated_requirements/decoupled-requirements.txt - the GPU lock minus the Google Cloud packages and the accelerator entries, kept current by a pre-commit hook - and fails if google.cloud.storage, google.api_core, google.auth, gcsfs, jetstream or the diagnostics clients are importable in it. Replaces the unreferenced jax 0.7.1 decoupled requirements file. Also fixes the wiring: the step sat between the previous step's run: and its env:, so Run Tests lost all ten INPUTS_* variables; it ran in all four cpu-unit shards; and $PYTHON_EXE was empty across the step boundary. Drops decoupled_target from the integration test and the four files pytest.ini ignores, where it never took effect, and corrects the marker docs. AI-Hypercomputer#4637 is a prerequisite; with it applied the suite is 46 passed, 2 skipped.
gulsumgudukbay
force-pushed
the
decoupled-mode-testing
branch
from
July 29, 2026 17:31
3cf898a to
a4ff38e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues #4642, which I was asked to take over. I don't have push access to that branch (it lives in this repo, not a fork), so this comes from the ROCm fork and carries @shralex's commits plus one on top addressing the review comments.
Summary
generated_requirements/decoupled-requirements.txtand fails the step ifgoogle.cloud.storage,google.api_core,google.auth,gcsfs,jetstreamor the diagnostics clients are importable in it.DECOUPLE_GCLOUD=TRUEonly changes whatgcloud_stubhands back, so a gate running in a venv that has google-cloud-storage installed would have passed Checkpoint_conversion: import GCS via gcloud_stub for decoupled mode #4003, [DECOUPLED-MODE] grpo_trainer: route goodput/vertex imports through gcloud_stub for decoupled #4180 and [DECOUPLED-MODE] checkpoint_conversion: import GCS via gcloud_stub in load_dynamic #4637.decoupled-requirements.txtis derived from the GPU pre-training lock minus the Google Cloud clients and the accelerator wheels, so the decoupled run uses the same versions as thegpu-unitjob (checked against agpu-unitrun on main, no version differences) and installs on any hardware. A decoupled run on an accelerator adds the dropped entries back via--print-accelerator-requirements. Deriving it from a lock rather than from base requirements keeps regeneration offline, which is what lets the newdecoupled-requirementspre-commit hook fail any requirements change that leaves it stale. Replacesrequirements_decoupled_jax_0_7.1.txt, which nothing referenced.run:and itsenv:, so Run Tests lost all tenINPUTS_*variables; it ran in all fourcpu-unitshards; and$PYTHON_EXEwas empty across the step boundary.decoupled_targetfrom the integration test and from the filespytest.iniignores, where the marker never took effect, and corrects the marker docs.Prerequisite
#4637 has to land first. Without it,
tests/unit/checkpointing_test.pycannot be collected once the Google Cloud packages are genuinely absent, and the whole runxits 2.Test plan
run:block as is underbash -eagainst a built wheel, with [DECOUPLED-MODE] checkpoint_conversion: import GCS via gcloud_stub in load_dynamic #4637 applied: 46 passed, 2 skipped, no collection errors.google-cloud-storagepresent it exits 1 and stops the step before pytest; same forgcsfs; a missing wheel fails loudly rather than skipping.DECOUPLE_GCLOUDunset fails at collection (ModuleNotFoundError: ml_goodput_measurement), so the gate can't pass trivially.pre-commit runpasses on every touched file, including the new hook.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.