[WIP][INFRA] Add smart class-level test selection for post-merge CI

[zhengruifeng]

What changes were proposed in this pull request?

This draft adds a standalone AI test selection post-merge workflow. It runs only for trusted Apache Spark pushes and does not modify the existing Build workflow or its reusable build_and_test.yml implementation.

Copilot CLI receives the changed paths and may inspect the checkout with read-only tools. It returns up to four Scala suite classes and four PySpark test modules. dev/ai_test_selection.py validates every target against the checkout, resolves Scala suites to their exact SBT project (including Connect, YARN, and Kafka subprojects), and rejects invalid or broad targets.

The workflow invokes every selected Scala suite separately with build/sbt "<project>/testOnly <Suite>". It builds Spark once when Python tests are selected, then invokes python/run-tests --testnames <module> separately for each selected PySpark test file. Copilot setup or request failures produce no selected targets and do not affect the existing CI workflows.

Why are the changes needed?

Module-level selection can execute many tests unrelated to a post-merge change. Class-level Scala selection and per-file PySpark selection provide a focused test signal while retaining the existing deterministic CI as independent coverage.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

• Reviewed the individual-test commands in https://spark.apache.org/developer-tools.html.
• Parsed the dedicated GitHub Actions workflow with PyYAML.
• Ran git diff --check.
• Validated known Scala, Connect, and PySpark targets with dev/ai_test_selection.py validate.
• No full Spark test suite was run; this workflow-only change will be exercised by post-merge CI.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)