Skip to content

ci: track test flavor in per-test baseline and rename job to Track Test Duration - #4659

Open
hsuan-lun-chiang wants to merge 1 commit into
mainfrom
track-test-duration-flavor
Open

ci: track test flavor in per-test baseline and rename job to Track Test Duration#4659
hsuan-lun-chiang wants to merge 1 commit into
mainfrom
track-test-duration-flavor

Conversation

@hsuan-lun-chiang

@hsuan-lun-chiang hsuan-lun-chiang commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR addresses follow-up feedback from b/528433969 by mitigating hardware-dependent test duration noise in the Per-Test Baseline comparison, excludes noisy CPU runners from macro-level dashboard alerting, and updates workflow job naming.

Problem & Context

  1. Previously, per_test_baseline.json tracked test durations using un-flavored test names. When a test ran on multiple hardware platforms with different execution times (e.g., QKClipMLATest.test_mla_dot_product_integration taking 0.94s on CPU vs 15.99s on GPU), baseline comparison falsely flagged regressions when comparing across hardware types.
  2. Macro-level dashboard tracking (github-action-benchmark) triggered false-positive CI alerts when total execution times on shared GitHub Actions CPU runners fluctuated (~25% variance due to multi-tenancy noise).

Solution

  1. Track Hardware Test Flavor (flavor::test_name): Baseline keys in per_test_baseline.json are now formatted as flavor::test_name (e.g., gpu-unit::..., tpu-unit::..., tpu7x-unit::...), isolating durations for each hardware flavor and device type.
  2. Exclude CPU Suites from Macro-Level Alerts: In process_test_results.py, CPU test suites (cpu-unit, cpu-torch-reference, cpu-post-training-unit) are skipped when generating benchmark-results.json. This prevents false-positive CI failure alerts from github-action-benchmark caused by shared CPU runner noise, while still maintaining individual CPU test baselines in per_test_baseline.json.
  3. Legacy Key Clean-up: Automatically purges un-flavored legacy baseline keys when initializing new_per_test_baseline.json.
  4. Rename Job & Workflow: Updated job name from Track Test Performance to Track Test Duration in track_performance.yml and ci_pipeline.yml.
  5. Security & Authentication: Pinned action references to full commit SHAs (zizmor compliance) and added explicit GITHUB_TOKEN remote authentication for pushing per_test_baseline.json to gh-pages when persist-credentials: 'false' is used.
  6. Unit Tests: Added tests/utils/process_test_results_test.py to verify per-flavor baseline isolation, regression checking, and CPU macro-benchmark exclusion.

Tests

Ran local unit tests and pre-commit hooks:

python3 tests/utils/process_test_results_test.py
pre-commit run --files tests/utils/process_test_results.py tests/utils/process_test_results_test.py .github/workflows/track_performance.yml .github/workflows/ci_pipeline.yml

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@hsuan-lun-chiang
hsuan-lun-chiang force-pushed the track-test-duration-flavor branch 6 times, most recently from 41df25b to da9f9f5 Compare July 30, 2026 10:57
@hsuan-lun-chiang
hsuan-lun-chiang marked this pull request as ready for review July 30, 2026 12:27
@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@hsuan-lun-chiang
hsuan-lun-chiang force-pushed the track-test-duration-flavor branch from da9f9f5 to 092d03c Compare July 31, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants