fix: run append loads at-most-once to prevent retry duplication - #44
Merged
Conversation
append is the only non-idempotent managed-table load mode. If the server commits the load but the response is lost (a timeout classified transient), _request_with_retry re-runs the append and duplicates the uploaded rows. Run append at-most-once; replace/delete/update/upsert are idempotent and keep the full retry budget. Also type `mode` as a literal of the accepted values so a bad mode string is caught at author time rather than as a server-side error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
load_managed_table(..., mode="append")no longer retries on transient errors.appendis the only non-idempotent load mode, so it now runs at most once;replace/delete/update/upsertare idempotent and keep the full retry budget unchanged.Also types
modeasLiteral["replace", "append", "delete", "update", "upsert"]on both the runtime and managed-client signatures, so a bad mode string is caught at author time instead of surfacing as a server-side error.Why
The retry wrapper re-runs the operation on any transient error, and the connector defaults to
max_retries=8. Withappendnow reachable, a load whose commit succeeded but whose response was lost (e.g. a gateway timeout, classified transient) gets re-run — appending the same uploaded rows again.replace/delete/update/upsertare naturally idempotent, soappendwas the only exposed mode with this hazard.The real long-term fix is server-side (append-dedup by upload/job in runtimedb, still pending); this is the self-contained SDK-side guard that closes the window now.
Scope
The behavioral change is local to
append._request_with_retrygainsretryable: bool = True(default preserves prior behavior), and only theappendcall site opts out — every other mode and every other operation is byte-identical to before.Ordering
Should merge before cutting
hotdata-framework 0.7.0, so the first release that shipsappendalso ships the guard (rather than needing a 0.7.1). The connector's existinghotdata-framework>=0.7.0floor already covers it — no floor change needed.Tests
test_append_load_runs_at_most_once—append+ transient error with budget 8 → called exactly once, not retried.test_idempotent_load_retries_on_transient—replace+ transient error with budget 3 → retried to the full budget.Verified locally:
uv sync --lockedresolves,pytest76 passed, ruff clean on touched files.