Skip to content

fix(arrow): return an error for incompatible Arrow export targets#8993

Open
robert3005 wants to merge 2 commits into
developfrom
claude/arrow-target-dtype-check
Open

fix(arrow): return an error for incompatible Arrow export targets#8993
robert3005 wants to merge 2 commits into
developfrom
claude/arrow-target-dtype-check

Conversation

@robert3005

@robert3005 robert3005 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Follow-up to #8902, which added to_arrow_array(arrow_type=...) and to_arrow(schema=...). Those parameters let a caller pick the target Arrow type, and execute_arrow_naive dispatches purely on that DataType without checking whether the source dtype can produce it.

#8902 added a guard for the byte arms. Five other arms have the same problem: they call execute::<T>, which is documented as panicking on a dtype mismatch:

/// Execute the array to canonical form and unwrap as a [`BoolArray`].
///
/// This will panic if the array's dtype is not bool.
impl Executable for BoolArray { ... }

So vx.array([1, 2, 3]).to_arrow_array(arrow_type=pa.bool_()) panics inside Canonical::into_bool() instead of returning an error. A panic crossing PyO3 becomes a PanicException, and in a DataFusion or DuckDB worker it is a panic rather than a surfaced error.

Measured before this change, exporting each source dtype to each target:

source\tgt      Utf8  LargeUtf8  Binary  LargeBinary  Utf8View  BinaryView  Int32  Int64  Float64  Boolean
utf8              ok         ok      ok           ok        ok         Err    Err    Err      Err    PANIC
binary            ok         ok      ok           ok       Err          ok    Err    Err      Err    PANIC
i32              Err        Err     Err          Err       Err         Err     ok     ok       ok    PANIC
f64              Err        Err     Err          Err       Err         Err     ok     ok       ok    PANIC
bool             Err        Err     Err          Err       Err         Err     ok     ok       ok       ok

Extending to the remaining arms, Boolean, Null, Decimal128, List, ListView, and FixedSizeList all panic. Date32, Timestamp, FixedSizeBinary, and Struct were already fine.

What changes are included in this PR?

A vortex_ensure! on the source dtype at the top of each affected arm, mirroring the existing check in to_arrow_byte_array: to_arrow_bool, to_arrow_null, to_arrow_decimal, to_arrow_list, to_arrow_list_view, and to_arrow_fixed_list.

The guards are per-arm rather than centralised because compatibility is not symmetric. A Bool source exports to Int32 through a cast kernel, but a Primitive source cannot produce Boolean. A single "type classes must match" rule at the dispatch point would reject conversions that work today, and the per-arm requirement is local and obvious.

Two rstest cases in vortex-arrow/src/executor/mod.rs:

  • incompatible_target_returns_error — 12 source/target pairs that previously panicked now return Err.
  • supported_cross_class_target_still_works — 7 legitimate cross-class conversions (bool → Int32, bool → Float64, i32 → Date32, i32 → Timestamp, utf8 → Binary, …) still succeed, pinning the asymmetry above so a future centralisation attempt cannot quietly break them.

What APIs are changed? Are there any user-facing changes?

No API changes. Behaviour change only on a path that previously aborted the process: requesting an Arrow type the source dtype cannot produce now returns a VortexError naming both the dtype and the target type.

Verification

  • cargo test -p vortex-arrow -p vortex-array -p vortex-datafusion — 3145 + 220 + 247 + 72 + 13 passed, 0 failed.
  • cargo clippy -p vortex-arrow --all-targets --all-features — clean.
  • cargo +nightly fmt --all, git diff --check — clean.

Not run: Python tests, docs doctests. This PR touches no Python or docs surface.

`execute_arrow_naive` dispatches purely on the requested Arrow `DataType`,
but the bool, null, decimal, list, list-view, and fixed-size-list arms then
call `execute::<T>`, which is documented as panicking when the array's dtype
does not match. Exporting an array to an incompatible Arrow type therefore
panicked instead of returning an error.

Guard each arm with `vortex_ensure!` on the source dtype, mirroring the
existing check in `to_arrow_byte_array`. The guards are per-arm rather than
centralised because compatibility is not symmetric: a Bool source exports to
Int32 via a cast kernel, while a Primitive source cannot produce Boolean, so
a single "type classes must match" rule would reject conversions that work.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
@codspeed-hq

codspeed-hq Bot commented Jul 27, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 1849 untouched benchmarks
⏩ 46 skipped benchmarks1


Comparing claude/arrow-target-dtype-check (cec62b9) with develop (b4f8d34)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
@robert3005
robert3005 force-pushed the claude/arrow-target-dtype-check branch from 1c03198 to cec62b9 Compare July 27, 2026 01:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants