Skip to content

[core] Fix query results being scrambled after querying $audit_log ba…#8804

Merged
JingsongLi merged 1 commit into
apache:masterfrom
L-Gryps:fix/audit-log-shared-options-leak
Jul 23, 2026
Merged

[core] Fix query results being scrambled after querying $audit_log ba…#8804
JingsongLi merged 1 commit into
apache:masterfrom
L-Gryps:fix/audit-log-shared-options-leak

Conversation

@L-Gryps

@L-Gryps L-Gryps commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Closes #8803

Purpose

When a primary-key table has 'table-read.sequence-number.enabled' = 'true', querying its $audit_log (or $binlog) system table and then querying the base table causes:

  • Column data to be shifted by one position (the internal _SEQUENCE_NUMBER value leaks into the first column, all real columns shift right), or
  • A hard ClassCastException when a base column is non-numeric, e.g. HeapLongVector cannot be cast to BytesColumnVector, surfaced by Flink as Recovery is suppressed by NoRestartBackoffTimeStrategy.

Root cause

AuditLogTable's constructor mutates the wrapped FileStoreTable's options map in place when sequence-number reading is enabled:

this.wrapped.options().put(CoreOptions.KEY_VALUE_SEQUENCE_NUMBER_ENABLED.key(), "true");

FileStoreTable.options() returns schema().options(), which is the same Map<String, String> reference held by the catalog-cached FileStoreTable. After the first SELECT * FROM t$audit_log, the cached base table permanently
carries key-value.sequence_number.enabled=true. Subsequent reads of the base table go through KeyValueTableRead.unwrap, which then makes ValueContentRowDataRecordIterator prepend a long _SEQUENCE_NUMBER column to every row.
The physical row now has one extra column ahead of the logical row type, causing column mis-alignment or vector type mismatch.

Fix

Instead of mutating the shared options map, create a private copy of the wrapped FileStoreTable that carries the internal KEY_VALUE_SEQUENCE_NUMBER_ENABLED=true option only for the audit-log's own reads. The catalog-cached
base table is left untouched.

wrapped = wrapped.copyWithoutTimeTravel(
        Collections.singletonMap(
                CoreOptions.KEY_VALUE_SEQUENCE_NUMBER_ENABLED.key(), "true"));

### Test

Added regression coverage at two layers:

- paimon-core/src/test/java/org/apache/paimon/table/system/AuditLogTableTest.javatestReadBaseTableAfterAuditLogDoesNotIncludeSequenceNumber: constructs an AuditLogTable and asserts the wrapped base table's row type /
  options are not polluted with _SEQUENCE_NUMBER / KEY_VALUE_SEQUENCE_NUMBER_ENABLED.

- paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/BatchFileStoreITCase.javatestReadBaseTableAfterAuditLogWithSequenceNumberEnabled: end-to-end reproduction of the reported user scenario (query
  t$audit_log, then re-query t); base-table result must remain unchanged.

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 93e047b into apache:master Jul 23, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants