Skip to content

[core] Support Deletion Vectors for Chain Table#8574

Open
juntaozhang wants to merge 4 commits into
apache:masterfrom
juntaozhang:support-chain-dv
Open

[core] Support Deletion Vectors for Chain Table#8574
juntaozhang wants to merge 4 commits into
apache:masterfrom
juntaozhang:support-chain-dv

Conversation

@juntaozhang

@juntaozhang juntaozhang commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Purpose

#8573

Tests

ChainSplitTest

  • testChainSplitSerdeWithDeletionFiles: verifies that ChainSplit can carry deletion files alongside data files and correctly serialize/deserialize them.

SplitSerializerTest

  • testV1Compatibility: ensures splits written before this change (without deletion files in ChainSplit) can still be deserialized.
  • Updates golden files from v1 to v2 and asserts that ChainSplit round-trips its deletion files, branch mappings, and bucket paths.

ChainTableFileStoreTableTest

  • testChainTableWithDelete: batch chain read with DELETE records, covering both deletion-vectors.enabled=true and false.
  • testChainTableWithUpdate: batch chain read with UPDATE records and DV enabled.
  • testChainTableCrossPartitionDeletionVectors: chain read with group partitions (chain-table.chain-partition-keys) and DV, verifying that deletion vectors are scoped per group.
  • testChainTableWithPartialUpdateDelete: confirms that PARTIAL_UPDATE + partial-update.remove-record-on-delete=true still works for chain tables without DV.
  • testChainTableRejectsPartialUpdateWithDeletionVectors: validates that enabling DV on a PARTIAL_UPDATE chain table is rejected.
  • testChainTableRejectsAggregate / testChainTableRejectsFirstRow: validates that AGGREGATE and FIRST_ROW merge engines are rejected for chain tables.

FlinkChainTableITCase

  • testChainTableWithDeletionVectors: end-to-end Flink batch SQL with DV enabled, including snapshot/delta deletes, updates, and compact_chain_table.
  • testStreamingReadWithDeletionVectors: Flink streaming read over a chain table with DV, covering the full-load phase (snapshot DV applied) and incremental phase (delta
    cross-partition delete/update changelog).

SparkChainTableITCase

  • testChainTableWithDeletionVectors: Spark batch SQL with group partitions and DV enabled, verifying cross-branch deletes and compact_chain_table.

@juntaozhang
juntaozhang marked this pull request as draft July 12, 2026 14:30
@juntaozhang
juntaozhang marked this pull request as ready for review July 23, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant