tests: resolve sam init menu positions dynamically in schemas tests - #9151
Open
roger-zhangg wants to merge 1 commit into
Open
tests: resolve sam init menu positions dynamically in schemas tests#9151roger-zhangg wants to merge 1 commit into
roger-zhangg wants to merge 1 commit into
Conversation
test_init_interactive_with_event_bridge_app_aws_schemas_python has been failing (see run 30657529336). It answered the runtime prompt with "8" and its comment said "8: Python 3.9", but the prompt offers 15 runtimes for the Infrastructure event management use case and position 8 is now java8.al2023. python3.9 has drifted to position 10. The test therefore selected a Java runtime and fed Python-flow answers into a Java prompt sequence: every later answer landed on the wrong question, and it finally asserted on hello_world_function/schema, a Python layout a Java project never produces. The positions moved when java8.al2023/java11.al2023/java17.al2023 were added (#9125). Nothing about the runtime list is stable, so hardcoding an index means any future runtime addition silently repoints these tests at a different runtime -- a failure that looks like a product bug rather than a stale fixture. Adds _get_runtime_position and _get_use_case_position, which look positions up from the same source the prompt uses (InitTemplates(). get_preprocessed_manifest plus get_sorted_runtimes), mirroring the existing _get_registry_position helper. All 8 tests in the file now resolve the template/use-case/runtime answers dynamically; the 4 Java tests and the go test were relying on the same brittle indexing and only passed by luck. Resolved positions are unchanged for Java (4) and go (1), and corrected from 8 to 10 for python3.9. Stale position comments updated to name the runtime rather than a number. Testing: make pr passes (9347 passed, 25 skipped, coverage 94.09%). Verified the helpers resolve to use case 8, java17.al2023 4, python3.9 10, go (provided.al2) 1, and rendered each test's user_input to confirm the answer sequences.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue(s) does this change fix?
N/A
Why is this change necessary?
test_init_interactive_with_event_bridge_app_aws_schemas_pythonis failing (run 30657529336).The test answered the runtime prompt with
8and its comment said# 8: Python 3.9. But the prompt offers 15 runtimes for the Infrastructure event management use case, and position 8 is nowjava8.al2023—python3.9has drifted to position 10:So the test selected a Java runtime and then fed Python-flow answers (
eb-app-python39, schema paginator choices) into a Java prompt sequence. Every later answer landed on the wrong question, and it finally asserted onhello_world_function/schema— a Python layout a Java project never produces.The positions moved when
java8.al2023/java11.al2023/java17.al2023were added (#9125).How does it address the issue?
Adds
_get_runtime_positionand_get_use_case_position, which resolve positions from the same source the prompt uses (InitTemplates().get_preprocessed_manifest(...)+get_sorted_runtimes) rather than hardcoding an index. This mirrors the existing_get_registry_positionhelper already in the file.All 8 tests now resolve template/use-case/runtime answers dynamically. The 4 Java tests and the go test were relying on the same brittle indexing and only passed by luck — a future runtime addition would have broken them the same way.
Resolved positions are unchanged for Java (4) and go (1), and corrected from 8 → 10 for
python3.9. Stale comments now name the runtime instead of a number.What side effects does this change have?
The helpers call
get_preprocessed_manifest, which these tests already exercise viasam inititself, so no new external dependency. If a runtime is ever removed from the manifest the helper raises with the available list, which fails loudly instead of silently selecting the wrong runtime.Note the same job also hit a separate pytest-xdist crash (
struct.error: 'i' format requires ... <= 2147483647— a worker tried to send a >2 GiB test report, killing the session and taking..._non_default_profile_selectionwith it). That is very likely a consequence of the misaligned answers producing runaway prompt output, so this fix should resolve it, but the 2 GiB report limit is a separate robustness issue not addressed here.Mandatory Checklist
PRs will only be reviewed after checklist is complete
make prpassesmake update-reproducible-reqsif dependencies were changedBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.