fix(arrow): return an error for incompatible Arrow export targets#8993
Open
robert3005 wants to merge 2 commits into
Open
fix(arrow): return an error for incompatible Arrow export targets#8993robert3005 wants to merge 2 commits into
robert3005 wants to merge 2 commits into
Conversation
`execute_arrow_naive` dispatches purely on the requested Arrow `DataType`, but the bool, null, decimal, list, list-view, and fixed-size-list arms then call `execute::<T>`, which is documented as panicking when the array's dtype does not match. Exporting an array to an incompatible Arrow type therefore panicked instead of returning an error. Guard each arm with `vortex_ensure!` on the source dtype, mirroring the existing check in `to_arrow_byte_array`. The guards are per-arm rather than centralised because compatibility is not symmetric: a Bool source exports to Int32 via a cast kernel, while a Primitive source cannot produce Boolean, so a single "type classes must match" rule would reject conversions that work. Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
This was referenced Jul 27, 2026
Merging this PR will not alter performance
Comparing Footnotes
|
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
robert3005
force-pushed
the
claude/arrow-target-dtype-check
branch
from
July 27, 2026 01:50
1c03198 to
cec62b9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Follow-up to #8902, which added
to_arrow_array(arrow_type=...)andto_arrow(schema=...). Those parameters let a caller pick the target Arrow type, andexecute_arrow_naivedispatches purely on thatDataTypewithout checking whether the source dtype can produce it.#8902 added a guard for the byte arms. Five other arms have the same problem: they call
execute::<T>, which is documented as panicking on a dtype mismatch:So
vx.array([1, 2, 3]).to_arrow_array(arrow_type=pa.bool_())panics insideCanonical::into_bool()instead of returning an error. A panic crossing PyO3 becomes aPanicException, and in a DataFusion or DuckDB worker it is a panic rather than a surfaced error.Measured before this change, exporting each source dtype to each target:
Extending to the remaining arms,
Boolean,Null,Decimal128,List,ListView, andFixedSizeListall panic.Date32,Timestamp,FixedSizeBinary, andStructwere already fine.What changes are included in this PR?
A
vortex_ensure!on the source dtype at the top of each affected arm, mirroring the existing check into_arrow_byte_array:to_arrow_bool,to_arrow_null,to_arrow_decimal,to_arrow_list,to_arrow_list_view, andto_arrow_fixed_list.The guards are per-arm rather than centralised because compatibility is not symmetric. A
Boolsource exports toInt32through a cast kernel, but aPrimitivesource cannot produceBoolean. A single "type classes must match" rule at the dispatch point would reject conversions that work today, and the per-arm requirement is local and obvious.Two
rstestcases invortex-arrow/src/executor/mod.rs:incompatible_target_returns_error— 12 source/target pairs that previously panicked now returnErr.supported_cross_class_target_still_works— 7 legitimate cross-class conversions (bool → Int32,bool → Float64,i32 → Date32,i32 → Timestamp,utf8 → Binary, …) still succeed, pinning the asymmetry above so a future centralisation attempt cannot quietly break them.What APIs are changed? Are there any user-facing changes?
No API changes. Behaviour change only on a path that previously aborted the process: requesting an Arrow type the source dtype cannot produce now returns a
VortexErrornaming both the dtype and the target type.Verification
cargo test -p vortex-arrow -p vortex-array -p vortex-datafusion— 3145 + 220 + 247 + 72 + 13 passed, 0 failed.cargo clippy -p vortex-arrow --all-targets --all-features— clean.cargo +nightly fmt --all,git diff --check— clean.Not run: Python tests, docs doctests. This PR touches no Python or docs surface.