ci: track test flavor in per-test baseline and rename job to Track Test Duration - #4659
Open
hsuan-lun-chiang wants to merge 1 commit into
Open
ci: track test flavor in per-test baseline and rename job to Track Test Duration#4659hsuan-lun-chiang wants to merge 1 commit into
hsuan-lun-chiang wants to merge 1 commit into
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
hsuan-lun-chiang
force-pushed
the
track-test-duration-flavor
branch
6 times, most recently
from
July 30, 2026 10:57
41df25b to
da9f9f5
Compare
hsuan-lun-chiang
marked this pull request as ready for review
July 30, 2026 12:27
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
hsuan-lun-chiang
requested review from
A9isha,
NuojCheng,
RissyRan,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
bvandermoon,
darisoy,
gagika,
gobbleturk,
hengtaoguo,
huytransformer,
igorts-git,
jiangjy1982,
khatwanimohit,
parambole,
richjames0,
shralex,
vipannalla and
xibinliu
as code owners
July 30, 2026 12:27
shralex
approved these changes
Jul 30, 2026
hsuan-lun-chiang
force-pushed
the
track-test-duration-flavor
branch
from
July 31, 2026 03:02
da9f9f5 to
092d03c
Compare
SurbhiJainUSC
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR addresses follow-up feedback from b/528433969 by mitigating hardware-dependent test duration noise in the Per-Test Baseline comparison, excludes noisy CPU runners from macro-level dashboard alerting, and updates workflow job naming.
Problem & Context
per_test_baseline.jsontracked test durations using un-flavored test names. When a test ran on multiple hardware platforms with different execution times (e.g.,QKClipMLATest.test_mla_dot_product_integrationtaking 0.94s on CPU vs 15.99s on GPU), baseline comparison falsely flagged regressions when comparing across hardware types.github-action-benchmark) triggered false-positive CI alerts when total execution times on shared GitHub Actions CPU runners fluctuated (~25% variance due to multi-tenancy noise).Solution
flavor::test_name): Baseline keys inper_test_baseline.jsonare now formatted asflavor::test_name(e.g.,gpu-unit::...,tpu-unit::...,tpu7x-unit::...), isolating durations for each hardware flavor and device type.process_test_results.py, CPU test suites (cpu-unit,cpu-torch-reference,cpu-post-training-unit) are skipped when generatingbenchmark-results.json. This prevents false-positive CI failure alerts fromgithub-action-benchmarkcaused by shared CPU runner noise, while still maintaining individual CPU test baselines inper_test_baseline.json.new_per_test_baseline.json.Track Test PerformancetoTrack Test Durationintrack_performance.ymlandci_pipeline.yml.zizmorcompliance) and added explicitGITHUB_TOKENremote authentication for pushingper_test_baseline.jsontogh-pageswhenpersist-credentials: 'false'is used.tests/utils/process_test_results_test.pyto verify per-flavor baseline isolation, regression checking, and CPU macro-benchmark exclusion.Tests
Ran local unit tests and pre-commit hooks:
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.