Expose metadata reading and writing in vortex-jni#8963
Open
robert3005 wants to merge 5 commits into
Open
Conversation
Signed-off-by: Robert Kruszewski <github@robertk.io>
mprammer
reviewed
Jul 25, 2026
Co-authored-by: Martin Prammer <martin@spiraldb.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
Merging this PR will improve performance by 19.59%
Performance Changes
Tip Curious why this is faster? Comment Comparing 
Footnotes
|
Follow-ups from review of the metadata JNI bindings: - `VortexWriter.Builder.build()` declared `throws IOException`, but every real failure arrived as the `RuntimeException` thrown across the JNI boundary, so a rejected metadata set escaped uncaught by `catch (IOException)`. Wrap native failures the way `writeBatch` and `finish` already do. This also routes the Spark writer's failures into its existing `IOException` handler, which logs the target path. - Cover `VortexWriteOptions::validate_metadata` in Rust: the empty-key and segment-count branches had no test in either language, and only the oversized-key branch was reachable from the Java suite. - Read metadata through the session footer cache. `readMetadata` opened the file with a bare `open_options()`, so reading metadata and then scanning the same file paid two footer reads. Factor the cached open out of `multi::open_file` into `multi::open_cached` and use it from both. Sources without a URI (the `NativeReadable` bridge) skip the cache, since they have no stable identity. - Release the per-entry local references in `extract_metadata`, matching `metadata_to_java` directly below it. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wt7vVoiJrjxHyDvy4XDjsW
`NativeReadable.name()` is already required to be stable and unique, and is what the scan path keys Java readables by, so a metadata read has an identity to cache under too. Pass it down and drop the "no key, no caching" path from `multi::open_cached`: every caller now supplies a fallback key, used when the source itself reports no URI. The name is normalized the same way `openFiles` normalizes it before registering readables, so a metadata read and a scan of the same readable share one cache entry rather than landing on two. Also pass the readable's known length, so a read that resolves against a cached footer validates it against the file size instead of trusting it blindly. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wt7vVoiJrjxHyDvy4XDjsW
robert3005
force-pushed
the
rk/jnimetadata
branch
from
July 26, 2026 16:19
b1b05ab to
1cd7164
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expose metadata reading and writing through jni bindings.
Change the VortexWrite into a builder to make it more convenient to add metadata