Skip to content

fix: run append loads at-most-once to prevent retry duplication - #44

Merged
shefeek-jinnah merged 1 commit into
mainfrom
fix/append-at-most-once
Jul 14, 2026
Merged

fix: run append loads at-most-once to prevent retry duplication#44
shefeek-jinnah merged 1 commit into
mainfrom
fix/append-at-most-once

Conversation

@shefeek-jinnah

Copy link
Copy Markdown
Contributor

What

load_managed_table(..., mode="append") no longer retries on transient errors. append is the only non-idempotent load mode, so it now runs at most once; replace/delete/update/upsert are idempotent and keep the full retry budget unchanged.

Also types mode as Literal["replace", "append", "delete", "update", "upsert"] on both the runtime and managed-client signatures, so a bad mode string is caught at author time instead of surfacing as a server-side error.

Why

The retry wrapper re-runs the operation on any transient error, and the connector defaults to max_retries=8. With append now reachable, a load whose commit succeeded but whose response was lost (e.g. a gateway timeout, classified transient) gets re-run — appending the same uploaded rows again. replace/delete/update/upsert are naturally idempotent, so append was the only exposed mode with this hazard.

The real long-term fix is server-side (append-dedup by upload/job in runtimedb, still pending); this is the self-contained SDK-side guard that closes the window now.

Scope

The behavioral change is local to append. _request_with_retry gains retryable: bool = True (default preserves prior behavior), and only the append call site opts out — every other mode and every other operation is byte-identical to before.

Ordering

Should merge before cutting hotdata-framework 0.7.0, so the first release that ships append also ships the guard (rather than needing a 0.7.1). The connector's existing hotdata-framework>=0.7.0 floor already covers it — no floor change needed.

Tests

  • test_append_load_runs_at_most_onceappend + transient error with budget 8 → called exactly once, not retried.
  • test_idempotent_load_retries_on_transientreplace + transient error with budget 3 → retried to the full budget.

Verified locally: uv sync --locked resolves, pytest 76 passed, ruff clean on touched files.

append is the only non-idempotent managed-table load mode. If the server
commits the load but the response is lost (a timeout classified transient),
_request_with_retry re-runs the append and duplicates the uploaded rows.
Run append at-most-once; replace/delete/update/upsert are idempotent and
keep the full retry budget.

Also type `mode` as a literal of the accepted values so a bad mode string
is caught at author time rather than as a server-side error.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct and well-scoped. append now runs at-most-once via the retryable opt-out while every other mode keeps its full retry budget, and the ManagedLoadMode literal is applied consistently. Both tests exercise the real behavior.

@shefeek-jinnah
shefeek-jinnah merged commit e3672cd into main Jul 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant