feat: NDJSON elements-file mode for partition (0.46.0)

[badGarnet]

What

Adds an opt-in NDJSON response mode to partition() that returns elements as a path to a file on disk instead of a parsed list, and ships it as 0.46.0.

from unstructured_client.general import PartitionAcceptEnum

res = client.general.partition(
    request=req,
    accept_header_override=PartitionAcceptEnum.APPLICATION_X_NDJSON,
)

try:
    with open(res.elements_file, encoding="utf-8") as f:
        for line in f:
            element = json.loads(line)
            ...
finally:
    os.unlink(res.elements_file)

PartitionResponse.elements_file is set instead of PartitionResponse.elements. The caller owns the file and must delete it. Requesting application/json remains the default and is entirely unchanged.

Why

On the split-PDF path the SDK rebuilt the whole document in memory in order to return it: a list per chunk, a flattened list, a json.dumps blob in create_response, and then the SDK's re-parse of that blob — four copies live at once, with the serialization step dominating peak usage. For documents with large metadata.image_base64 payloads this is the difference between a job completing and being OOM-killed.

In the new mode the per-chunk temp files are concatenated on disk and never parsed, so peak memory is roughly one chunk rather than the whole document.

How

• combine_chunk_files_to_ndjson concatenates chunk files on disk. Each chunk is sniffed for its first non-whitespace character, so a server returning application/json still works; chunks that are already NDJSON are copied through without parsing.
• ndjson_mode depends only on the Accept header, never on split_pdf_cache_tmp_data. Those are set by different parties, so gating on both let them disagree — the server would return NDJSON while the hook took the JSON path and res.json() raised on a body this client had itself requested.
• Both caching modes are handled. A cached chunk contributes its existing temp-file path; an uncached one spills its body verbatim and then releases it, since every response is retained in api_successful_responses and leaving _content set would keep the document resident regardless.
• The combined file is deliberately written outside the operation's TemporaryDirectory, which _clear_operation removes as soon as after_success returns.

Temp-file ownership

Everything this path creates is accounted for:

• Spilled chunk bodies are written inside the operation's temp directory and unlinked once combined.
• The combined file is deleted when a chunk failure means it is never handed back to the caller.
• Recombination writes to a staging file that is atomically renamed into place only on success, so a malformed chunk cannot orphan a partial file.
• No combined file is created at all when every chunk failed.

Security

The elements-file marker is an httpx response extension, not a response header. Extensions are populated by the transport, so a remote server cannot set the key. A header would be wire-controlled, and since callers are documented to open elements_file and then delete it, that would hand a hostile server an arbitrary local file to destroy. A real server body is always copied to a file this client creates.

Regeneration

elements_file is client-side only and can never come from the OpenAPI spec, so a regeneration would silently drop it. Both general.py and models/operations/partition.py are now in .genignore, and test_regeneration_guards.py fails if either entry is lost.

Known limitation

For unsplit inputs (non-PDF, or split_pdf_page=False) the response body is still fully read into memory before being written to disk, so peak is 2x the body rather than bounded. This does not affect the split-PDF path that motivated the change, where the SDK only ever sees an empty dummy response or the small synthetic path response. Fixing it means stream=True for NDJSON requests, which makes raw_response.content raise on the returned closed response — a user-visible change worth its own review. Tracked separately.

Testing

• New _test_unstructured_client/unit/test_ndjson_elements_file.py — recombination across JSON-array / NDJSON / mixed chunk formats, order preservation, byte-exact payload round-trip, non-ASCII, temp-file lifecycle on success and failure, and regression guards for the header-spoofing and partial-output defects.
• 235 unit tests and 64 contract tests pass; pylint 10.00/10; mypy clean.

🤖 Generated with Claude Code

[cubic-dev-ai]

All reported issues were addressed across 12 files

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/unstructured_client/_hooks/custom/request_utils.py">

<violation number="1" location="src/unstructured_client/_hooks/custom/request_utils.py:363">
P3: The new unlink-on-write-failure cleanup in write_chunk_body_to_temp — the entire point of this hardening — has no test coverage: the existing tests only exercise the success roundtrip. Consider adding a regression test that forces the write to raise (monkeypatch the file object's write or response.content) and asserts the temp file no longer exists and the exception still propagates, so the orphan-prevention behavior is pinned.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: The new unlink-on-write-failure cleanup in write_chunk_body_to_temp — the entire point of this hardening — has no test coverage: the existing tests only exercise the success roundtrip. Consider adding a regression test that forces the write to raise (monkeypatch the file object's write or response.content) and asserts the temp file no longer exists and the exception still propagates, so the orphan-prevention behavior is pinned.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/unstructured_client/_hooks/custom/request_utils.py, line 363:

<comment>The new unlink-on-write-failure cleanup in write_chunk_body_to_temp — the entire point of this hardening — has no test coverage: the existing tests only exercise the success roundtrip. Consider adding a regression test that forces the write to raise (monkeypatch the file object's write or response.content) and asserts the temp file no longer exists and the exception still propagates, so the orphan-prevention behavior is pinned.</comment>

<file context>
@@ -360,8 +360,18 @@ def write_chunk_body_to_temp(response: httpx.Response, dir_: Optional[str] = Non
     fd, path = tempfile.mkstemp(suffix=".ndjson", dir=dir_ or tempfile.gettempdir())
-    with os.fdopen(fd, "wb") as f:
-        f.write(response.content)
+    try:
+        with os.fdopen(fd, "wb") as f:
+            f.write(response.content)
</file context>

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="_test_unstructured_client/unit/test_ndjson_elements_file.py">

<violation number="1" location="_test_unstructured_client/unit/test_ndjson_elements_file.py:225">
P3: The orphan assertion in both copy-failure tests (`tmp_path.glob("unst_elements_*") == []`) only proves cleanup if the partial file actually landed in tmp_path — which is guaranteed solely by `_new_elements_file()` honoring the monkeypatched global `tempfile.tempdir` (it passes no explicit `dir`). The tests never verify that setup assumption, so a future change that stops routing the temp file through the global tempdir would make both tests pass vacuously while leaking a partial file into the real temp dir. Consider recording the path production actually created during the failing copy (e.g., patching `_new_elements_file` or capturing `out.name`) and asserting it was under tmp_path, so the cleanup check can't become vacuous.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: The orphan assertion in both copy-failure tests (tmp_path.glob("unst_elements_*") == []) only proves cleanup if the partial file actually landed in tmp_path — which is guaranteed solely by _new_elements_file() honoring the monkeypatched global tempfile.tempdir (it passes no explicit dir). The tests never verify that setup assumption, so a future change that stops routing the temp file through the global tempdir would make both tests pass vacuously while leaking a partial file into the real temp dir. Consider recording the path production actually created during the failing copy (e.g., patching _new_elements_file or capturing out.name) and asserting it was under tmp_path, so the cleanup check can't become vacuous.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At _test_unstructured_client/unit/test_ndjson_elements_file.py, line 225:

<comment>The orphan assertion in both copy-failure tests (`tmp_path.glob("unst_elements_*") == []`) only proves cleanup if the partial file actually landed in tmp_path — which is guaranteed solely by `_new_elements_file()` honoring the monkeypatched global `tempfile.tempdir` (it passes no explicit `dir`). The tests never verify that setup assumption, so a future change that stops routing the temp file through the global tempdir would make both tests pass vacuously while leaking a partial file into the real temp dir. Consider recording the path production actually created during the failing copy (e.g., patching `_new_elements_file` or capturing `out.name`) and asserting it was under tmp_path, so the cleanup check can't become vacuous.</comment>

<file context>
@@ -171,6 +178,83 @@ def test_write_chunk_body_to_temp_roundtrips(tmp_path):
+    The destination is created with delete=False so it can outlive the helper, which is
+    exactly what makes an interrupted copy leak.
+    """
+    monkeypatch.setattr(tempfile, "tempdir", str(tmp_path))
+
+    class _FailingResponse:
</file context>