Skip to content

fix: skip byte arrays in shadow mismatch JSON - #116

Closed
lan17 wants to merge 1 commit into
mainfrom
fix/shadow-log-skip-binary
Closed

fix: skip byte arrays in shadow mismatch JSON#116
lan17 wants to merge 1 commit into
mainfrom
fix/shadow-log-skip-binary

Conversation

@lan17

@lan17 lan17 commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Summary

Follow-up to #113. Byte arrays are the one shape left where the mismatch JSON preview is actively counterproductive.

Native JSON expands a Buffer or other Uint8Array view into one decimal element per byte, so an 8 KiB-clamped preview becomes a run of digits:

{"blob":{"type":"Buffer","data":[7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,...

That is ~6 characters of JSON per byte of payload: the worst cost-per-byte of any shape, and it crowds every other field out of the capped preview. Measured before this change, a record with a 200 KB Buffer field produced ~2 MB of intermediate JSON to emit 8192 characters of the digit 7.

After, at any depth:

{"id":"1","blob":"<binary 200000 bytes>"}

Implementation

Buffer.prototype.toJSON runs before a JSON.stringify replacer, which splits the handling:

  • A plain Uint8Array view reaches the replacer intact and is swapped for the marker before expansion, so no per-byte array is built.
  • A Buffer has already been converted to { type: "Buffer", data: [...] } by the time the replacer sees it, so it is recognized from that result. This keeps the array out of the intermediate JSON string, but the toJSON allocation itself is still paid — the README notes that limit rather than implying otherwise.
  • A top-level view is checked before JSON.stringify so a bare binary cached value skips toJSON entirely.

A plain record already shaped like { type: "Buffer", data: [...] } is reduced the same way. That is deliberate: native JSON renders a real Buffer identically, so collapsing the shape loses no information the previous output preserved.

Scope

No public API change. shadow-log-json.ts is internal and not exported from the root entry, so this alters only the contents of an opt-in logMismatches warning. No new config, metric, label, or dependency. The 2 KiB key cap, 8 KiB value caps, ...[truncated] marker, and null-on-unserializable behavior are untouched.

Validation

Node 26.5.0, from main@34bd022:

  • pnpm typecheck — clean
  • pnpm test — 409 passed (4 new cases covering nested and top-level views, offset/pooled views, large-buffer crowding, and the look-alike record)
  • pnpm build — clean; verified the marker is present in dist/index.js and the CommonJS bundle loads
  • pnpm test:package — not run locally (corepack unavailable in my environment); it exercises the packed public surface, which this change does not touch. Worth confirming in CI.

🤖 Generated with Claude Code

Native JSON expands a Buffer or other Uint8Array view into one decimal
element per byte, so an 8 KiB-clamped mismatch preview became a run of
digits with no diagnostic value, and a 200 KB view produced roughly 2 MB
of intermediate JSON to do it.

Replace byte arrays with a "<binary N bytes>" marker at any depth. Plain
views are intercepted by the replacer before expansion; Buffer runs its
own toJSON first, so it is recognized from that result instead.
@lan17
lan17 force-pushed the fix/shadow-log-skip-binary branch from 6cee172 to 8c3115b Compare July 31, 2026 22:39
@lan17 lan17 closed this Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant