Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .genignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,14 @@ src/unstructured_client/general.py
# Custom min_attempts / absolute_max_elapsed_time_ms fields on BackoffStrategy.
# Push upstream to Speakeasy templates to remove this entry.
src/unstructured_client/utils/retries.py

# Custom elements_file field on PartitionResponse, for the NDJSON elements-file mode.
# The field is client-side only - the server never returns it - so it cannot come from
# the OpenAPI spec, and regenerating would drop it. If /general/v0/general gains a new
# response field, follow the same procedure as general.py above.
# See test_regeneration_guards.py::test_partition_response_keeps_elements_file.
src/unstructured_client/models/operations/partition.py

# Docs for that same custom elements_file field. This file is generated from the spec
# and the daily generation workflow would otherwise drop the row on its next run.
docs/models/operations/partitionresponse.md
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
## 0.46.0

### Features
* Add an NDJSON elements-file mode to `partition()`. Pass `accept_header_override=PartitionAcceptEnum.APPLICATION_X_NDJSON` to get `PartitionResponse.elements_file` — a path to an NDJSON file with one element per line — instead of `PartitionResponse.elements`. On the split-PDF path the per-chunk temp files are concatenated on disk rather than parsed, flattened, re-serialized with `json.dumps` and re-parsed by the SDK, which held four copies of the document in memory at once. Peak memory becomes roughly one chunk instead of the whole document. **The caller owns the returned file and is responsible for deleting it.** Requesting `application/json` (the default) is unchanged.

## 0.45.0

### Features
Expand Down
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -427,6 +427,37 @@ req = operations.PartitionRequest(
)
```

### Streaming elements to a file instead of memory

For very large documents, the parsed element list can dominate the client's memory: the split-PDF path holds a list per chunk, a flattened list, a serialized blob and the SDK's re-parse of that blob. Request `application/x-ndjson` to skip all of it. The chunk responses are concatenated on disk and you get back a path in `elements_file` instead of a list in `elements`, which keeps peak memory at roughly one chunk.
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.

**You own the returned file and are responsible for deleting it.**

> [!NOTE]
> The memory saving applies to the split-PDF path, i.e. a PDF with `split_pdf_page=True` (the default). For unsplit inputs — a non-PDF file, or `split_pdf_page=False` — the response body is still read fully into memory before being written to disk, so peak memory can reach roughly twice the body size. You still get `elements_file` either way.

Example:
```python
import json
from pathlib import Path

from unstructured_client.general import PartitionAcceptEnum

res = client.general.partition(
request=req,
accept_header_override=PartitionAcceptEnum.APPLICATION_X_NDJSON,
)

try:
with open(res.elements_file, encoding="utf-8") as f:
for line in f:
element = json.loads(line)
...
finally:
# missing_ok so a failure to open the file isn't masked by the cleanup.
Path(res.elements_file).unlink(missing_ok=True)
```

<!-- Start File uploads [file-upload] -->
## File uploads

Expand Down
10 changes: 10 additions & 0 deletions RELEASES.md
Original file line number Diff line number Diff line change
Expand Up @@ -1241,3 +1241,13 @@ Based on:
- [python v0.45.0] .
### Releases
- [PyPI v0.45.0] https://pypi.org/project/unstructured-client/0.45.0 - .

## 2026-08-01 00:00:00
### Changes
Based on:
- OpenAPI Doc
- Speakeasy CLI 1.601.0 (2.680.0) https://github.com/speakeasy-api/speakeasy
### Generated
- [python v0.46.0] .
### Releases
- [PyPI v0.46.0] https://pypi.org/project/unstructured-client/0.46.0 - .
Loading