feat: stream large Parquet uploads via presigned session API (0.7.1) - #46
Conversation
Replaces the full-file f.read() in upload_parquet() with the presigned upload session API, so only one part_size chunk (not the entire file) is in memory at a time. Falls back to POST /v1/files when the server returns 501 (backends without presigned URL support). Bumps to 0.7.1.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| collected.append( | ||
| FinalizeUploadPart( | ||
| part_number=i + 1, | ||
| e_tag=resp.headers["ETag"], |
There was a problem hiding this comment.
super nit: resp.headers["ETag"] raises a bare KeyError if a storage backend/proxy omits the ETag on a 2xx PUT, which would surface as an opaque traceback rather than the clean RuntimeError used for the other failure paths here. Consider guarding it, e.g. etag = resp.headers.get("ETag") and raising a descriptive error if missing. (not blocking)
| part_urls=None, | ||
| url=None, | ||
| ) | ||
| return SimpleNamespace(mode=mode, **{**defaults, **kw}) |
There was a problem hiding this comment.
nit: the session/finalized responses are faked with SimpleNamespace, so the attribute names the production code depends on (session.mode, session.url, session.part_urls, session.part_size, session.headers, session.finalize_token, finalized.upload_id) are never checked against the real generated hotdata response models. If those model field names drift from these assumptions, the tests still pass but the upload breaks at runtime. Worth at least one assertion that constructs the real response model, or a comment noting the field names are pinned to the SDK version. (not blocking)
1 participant
Summary
f.read()inupload_parquet()with the presigned upload session API (POST /v1/uploads), so only onepart_sizechunk is resident in memory at a time during large Parquet loadsmultipartmode: iterates overpart_urls, reading and PUTting one chunk per URL, collecting ETags for finalizesinglemode: reads the whole file (small files do not need chunking) and does a single PUTPOST /v1/filespath if the server returns 501 (storage backends without presigned URL support)urllib3.PoolManagerdirectly (already in the dependency chain -- no new deps)0.7.1Context
Root cause of the jaffle_shop Snowflake ingest OOM:
upload_parquet()calledf.read()on each Parquet file before uploading, so the entire file lived in memory. Intermediate mitigation (DATA_WRITER__FILE_MAX_BYTES=200MBin dlthubworker) limits file size, but loading still peaks at 200MB per file. This fix eliminates that peak.After this ships: bump
hotdata-dlt-destinationlockfile and publish 0.9.2, then rebuild the dlthubworker Docker image. Once deployed, the 200MB cap iningest_job.pycan be loosened or removed.