Skip to content

[core] Restore shredding write plans from historical files (variant & map shredding)#8827

Open
lxy-9602 wants to merge 3 commits into
apache:masterfrom
lxy-9602:map-variant-restore
Open

[core] Restore shredding write plans from historical files (variant & map shredding)#8827
lxy-9602 wants to merge 3 commits into
apache:masterfrom
lxy-9602:map-variant-restore

Conversation

@lxy-9602

@lxy-9602 lxy-9602 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Purpose

This is a subtask of MAP shared-shredding support.

This PR introduces a shared history mechanism for restoring per-file shredding write plans from recent data files.

  • Add format-independent ShreddingFileMetadata, ShreddingWritePlanHistory, and ShreddingWritePlanPreparation abstractions.
  • Lazily load physical schemas and field metadata from recent restored files, while preserving schema evolution and column renames through field IDs.
  • Update history after each rolling file so subsequent files can use the latest metadata.
  • Restore MAP shared-shredding column counts from historical maximum row widths. Files without valid history continue to use the configured max-columns.
  • Add the opt-in variant.restoreShreddingSchemaFromHistory option. Existing Variant inference behavior remains unchanged by default; missing Variant fields still fall back to row sampling.
  • Propagate history through append-only, primary-key, changelog, thin-mode, and compaction/rewrite writers. Rewrite outputs derive history from their actual input files.
  • Support metadata recovery through both ORC and Parquet, including mixed-format history.

Tests

Over 1,900 lines of this PR are dedicated to comprehensive test coverage.

Added or updated coverage for:

  • Shredding history ordering, file limit, schema-ID normalization, column renames, and recoverable metadata IO failures.
  • MAP adaptive column-count restoration, multiple independent MAP fields, rolling files, mixed ORC/Parquet history, and partition/bucket isolation.
  • Variant schema restoration, nested Variant fields, partial restoration, and fallback inference for newly added Variant fields.
  • Append-only and primary-key tables, including thin mode, aggregation, changelog files, external spill, deletion vectors, and regular/partial/clustering compaction.
  • ORC and Parquet metadata round trips and per-file history updates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant