Antalya-26.3 - Backport flaky-fix commits from upstream (2026-07-27) - #2117
Open
github-actions[bot] wants to merge 4 commits into
Open
Antalya-26.3 - Backport flaky-fix commits from upstream (2026-07-27)#2117github-actions[bot] wants to merge 4 commits into
github-actions[bot] wants to merge 4 commits into
Conversation
The test injects a 1200ms network delay and expects an S3 INSERT to time out and raise. It set no explicit S3 timeout, so raising relied on a default timeout being below 1200ms on the connection path actually taken. On a fresh connection the adaptive connect timeout (500ms first attempt, 1000ms on retries) trips reliably. But when the S3 request reuses a pooled keep-alive connection there is no fresh TCP connect, so the connect timeout does not apply; the only remaining gate is the send/receive idleness timeout, whose default (adaptive 3000ms / configured 30000ms) is far above 1200ms, so the write completes and pytest.raises sees no exception. That is the rare DID NOT RAISE flake (one occurrence in 40 days of CI, zero on master). Make the request timeout the single failure mechanism: disable adaptive timeouts, keep the connect timeout above the delay, and set s3_request_timeout_ms=500. The send/receive idleness timeout applies to every attempt on both fresh and reused connections, so the 1200ms delay always exceeds it and the write times out deterministically. Also assert the raised error is a timeout, so the test cannot pass on an unrelated error. (cherry picked from commit bf0f909)
The first query materialized a ~1.3 GB haystack column (numbers(1023), each row up to 2.66 MB) only to trip the per-row regexp_max_matches_per_row (default 1000) fast-throw TOO_LARGE_ARRAY_SIZE. The throw fires on the first row (2600 matches); the rest of the column is wasted allocation. Under the shared parallel-runner memory budget, building it could hit MEMORY_LIMIT_EXCEEDED before the expected throw, so the test intermittently failed (0 master failures, 27 unrelated PRs in 30 days). Use a single materialized 1001-character row, which crosses the same per-row limit at ~1 KB instead of 1.3 GB. materialize() keeps the runtime code path the throw lives in. A low max_memory_usage on the query also guards against regressing to a large fixture. The second query is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit b54dc15)
The per-format setup produced all messages (including the tail broken message) to the topic before the destination materialized views existed, then created the Kafka table, the data view and the errors view in sequence. Creating the first materialized view starts the streaming loop, so on slow builds the loop could consume and commit the broken message before the errors view was attached. The broken message is then past the committed offset and never reaches the errors view: the data view has its rows but the errors view stays empty, failing the test at "Error row for format X did not appear in kafka_errors_X_mv". Create both materialized views, detach/re-attach the Kafka table, and only then produce the messages, so nothing is consumable until both views exist. This mirrors the fix accepted for the sibling test test_kafka_engine_put_errors_to_stream in ClickHouse#107025. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit 231c36c)
test_async_connect verifies async-connect preemption: the first replica of
each shard in test_cluster_connect is an unreachable address (129.0.0.1 /
129.0.0.2), so the in-progress connect must be preempted by the
hedged_connection_timeout_ms timer and switch to the good replica. That timer
is the only path counted by the HedgedRequestsChangeReplica event, which the
test asserts fired (>= 2, one per shard).
The connect to an unreachable 129.0.0.x address does not always stall.
Depending on the runner's routing it can fail fast (host unreachable) before
the 100 ms timer. A fast failure switches the replica through the
connection-failure path (ConnectionEstablisher), which increments
DistributedConnectionFailTry, not HedgedRequestsChangeReplica. When both shards
fail fast the event stays 0; system.events omits zero-valued events, so the
query returns an empty string and int('') raises "invalid literal for int()
with base 10: ''". Slow builds (ASan, MSan, coverage) hit the fast-fail path
often enough to flake while master stays green (15 failures across 7 unrelated
PRs over 90 days, first 2026-05-05, 0 on master).
Drop the initiator's outbound packets to 129.0.0.1/129.0.0.2 with
PartitionManager so the connect always stalls and is preempted by the hedged
timer, taking the counted path. This keeps the original assertion and the
async-connect-preemption intent, and makes it deterministic regardless of
routing. The engine is not changed: hedged failover already reaches the good
replica on both paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit c1e3273)
github-actions
Bot
added
antalya
cicd
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated backport of upstream flaky-fix commits.
Applied
bf0f90973919Fix flaky test_s3_table_functions_timeouts (committed 2026-07-23T01:36:26Z)b54dc15ccf10Fix flaky test 01661_extract_all_groups_throw_fast under memory pressure (committed 2026-07-23T02:50:38Z)231c36ce98d6Fix flaky test_kafka_formats_with_broken_message (committed 2026-07-23T22:46:53Z)c1e3273140e6Fix flaky test_hedged_requests::test_async_connect (committed 2026-07-24T17:38:27Z)Skipped (cherry-pick conflict — manual backport needed)
2d531c66b031Fix flaky test04538_with_fill_max_execution_time(committed 2026-07-21T02:39:18Z)f2bf53742903Fix flaky test 04011_predicate_statistics_log under parallel replicas (committed 2026-07-23T12:07:23Z)9bcba1b43879Fix flaky test 04546_constant_join_shrink_stored_blocks (committed 2026-07-25T20:57:39Z)