Support TsFile properties and optional table point counting by jt2594838 · Pull Request #882 · apache/tsfile

jt2594838 · 2026-07-29T10:41:58Z

Summary

add TsFileProperties read/write APIs to TsFileIOWriter, TsFileSequenceReader, ITsFileWriter, and ITsFileReader
add an optional TsFileIOWriter mode that records non-null FIELD point counts per table in TsFile properties
add a repair tool that detects and backfills missing table point-count properties in existing complete TsFiles
package Linux and Windows launch scripts and document the tool in the Java tools README
add API, repair-tool, and performance-test coverage

Point-count semantics

properties use the tablePointCount.<tableName> key format
timestamps, TAG values, and null FIELD values are excluded
collection is disabled by default

Table point-count repair tool

scans table-model metadata and calculates missing point counts without enumerating devices
preserves existing data and unrelated TsFile properties
leaves files unchanged when all table point-count properties are already present
rejects missing or incomplete TsFiles
writes and flushes a temporary file before replacing the source
uses reflink copy-on-write on supported Linux and macOS file systems, with a full prefix-copy fallback
provides tsfile-table-point-count.sh and tsfile-table-point-count.bat

Verification

mvn -P with-java -pl java/tsfile -am test-compile -DskipTests
repair-tool tests cover multi-table counting, null FIELD exclusion, property preservation, file completeness, and idempotent updates
disabled mode: PID 7620, 36,350 bytes, 67.309 ms
enabled mode: PID 25216, 36,410 bytes, 74.772 ms
file-size overhead: 60 bytes (0.165%)

Each performance mode performed an independent warm-up in its own Surefire JVM.

codecov-commenter · 2026-07-29T11:05:58Z

Codecov Report

❌ Patch coverage is 71.22642% with 61 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.57%. Comparing base (94f6dac) to head (a095b07).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
...apache/tsfile/utils/TsFileTablePointCountTool.java	70.27%	44 Missing ⚠️
...rg/apache/tsfile/file/metadata/TsFileMetadata.java	60.00%	14 Missing ⚠️
...org/apache/tsfile/write/writer/TsFileIOWriter.java	88.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #882      +/-   ##
===========================================
+ Coverage    60.47%   60.57%   +0.10%     
===========================================
  Files          744      745       +1     
  Lines        49021    49267     +246     
  Branches      7787     7830      +43     
===========================================
+ Hits         29644    29844     +200     
- Misses       17972    18021      +49     
+ Partials      1405     1402       -3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jt2594838 · 2026-07-30T08:23:05Z

  }

+  /** Get the properties stored in the TsFile metadata. */
+  public Map<String, String> getTsFileProperties() throws IOException {


This exposes TsFile properties through the low-level reader API so callers can retrieve file-level metadata without depending on internal metadata objects. The corresponding V4 reader/writer interfaces expose the same capability across implementations while preserving existing behavior for callers that do not use properties.

jt2594838 · 2026-07-30T08:23:06Z

  public void endCurrentChunk() {
    this.currentChunkMetadataSize += currentChunkMetadata.getRetainedSizeInBytes();
    chunkMetadataCount++;
+    if (recordTablePointCount


Point counting is opt-in to avoid changing existing write-path cost or file output by default. It aggregates only non-null FIELD values into tablePointCount.<tableName> properties; the mask check excludes timestamp chunks, while table-model device detection excludes tree-model data.

jt2594838 · 2026-07-30T08:23:08Z

+   * untouched. The source file is replaced only after a complete rewritten copy has been forced to
+   * disk.
+   */
+  public static UpdateStatus updateTablePointCountIfMissing(File file) throws IOException {


This entry point backfills missing counts for complete table-model TsFiles by scanning timeseries metadata instead of enumerating every device. It preserves unrelated properties, skips files that already have valid counts, and replaces the source only after a temporary rewrite has been flushed; reflink cloning is used where available with a prefix-copy fallback.

jt2594838 · 2026-07-30T08:23:10Z

            <destName>tools/arrow2tsfile.bat</destName>
        </file>
+        <file>
+            <source>${maven.multiModuleProjectDirectory}/java/tools/src/assembly/resources/tools/tsfile-table-point-count.sh</source>


Packaging both launchers makes the repair utility directly available from the assembled tools distribution. The shell entry is explicitly marked executable, and the Windows launcher is placed beside it using the same tool name.

jt2594838 · 2026-07-30T08:23:11Z

 - Multiple batches: `{source_basename}_1.tsfile`, `{source_basename}_2.tsfile`, ...
 - Table name and output filename are independent — table name comes from schema or `--table_name`, filename comes from source file.

+## Table Point Count Tool


This section documents the tool point-count semantics, invocation, result statuses, in-place replacement behavior, and disk-space and backup considerations. The matching Chinese README and message bundles keep the user experience consistent with the project localization convention.

jt2594838 · 2026-07-30T08:23:13Z

+   * file data, and leave the file byte-for-byte unchanged when run again.
+   */
+  @Test
+  public void backfillMissingTablePointCountProperties() throws Exception {


This test covers multi-table counting, null FIELD exclusion, preservation of unrelated properties and file data, and idempotent re-execution. The related API and writer tests cover property round trips and opt-in collection, while the performance test isolates enabled and disabled modes in separate JVMs to avoid warm-up bias.

Copilot

Pull request overview

Adds TsFile metadata property read/write support across writer/reader APIs, introduces optional table-level non-null FIELD point-count recording, and provides a repair tool to backfill missing table point-count properties in existing complete TsFiles.

Changes:

Add TsFileProperties APIs to V4 writer/reader interfaces and to TsFileSequenceReader / TsFileIOWriter.
Add optional TsFileIOWriter mode to record per-table non-null FIELD point counts into TsFile properties.
Add TsFileTablePointCountTool (plus scripts, i18n messages, and tests) to detect/backfill missing table point-count properties.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
java/tsfile/src/test/java/org/apache/tsfile/write/TsFileWriteApiTest.java	Adds a correctness test for table point-count property semantics (non-null FIELD only).
java/tsfile/src/test/java/org/apache/tsfile/write/TablePointCountPerformanceTest.java	Adds an opt-in (property-gated) performance comparison test for point counting.
java/tsfile/src/test/java/org/apache/tsfile/utils/TsFileTablePointCountToolTest.java	Adds repair-tool coverage: scanning, update behavior, property preservation, idempotency.
java/tsfile/src/test/java/org/apache/tsfile/read/TsFileV4ReadWriteInterfacesTest.java	Verifies properties are visible via both `TsFileSequenceReader` and V4 `ITsFileReader`.
java/tsfile/src/main/java/org/apache/tsfile/write/writer/TsFileIOWriter.java	Adds property collection and optional table point-count recording at write time.
java/tsfile/src/main/java/org/apache/tsfile/write/v4/ITsFileWriter.java	Adds `addTsFileProperty` to the V4 writer API.
java/tsfile/src/main/java/org/apache/tsfile/write/v4/AbstractTableModelTsFileWriter.java	Implements `addTsFileProperty` by delegating to the underlying writer.
java/tsfile/src/main/java/org/apache/tsfile/utils/TsFileTablePointCountTool.java	Introduces the repair tool: scan metadata and rewrite metadata tail (reflink+fallback).
java/tsfile/src/main/java/org/apache/tsfile/read/v4/ITsFileReader.java	Adds `getTsFileProperties` to the V4 reader API.
java/tsfile/src/main/java/org/apache/tsfile/read/v4/DeviceTableModelReader.java	Implements `getTsFileProperties` by delegating to the underlying reader.
java/tsfile/src/main/java/org/apache/tsfile/read/TsFileSequenceReader.java	Adds `getTsFileProperties()` accessor for metadata properties.
java/tools/src/assembly/tools.xml	Packages the new tool launch scripts into the tools distribution.
java/tools/src/assembly/resources/tools/tsfile-table-point-count.sh	Adds Linux/macOS launcher for the repair tool.
java/tools/src/assembly/resources/tools/tsfile-table-point-count.bat	Adds Windows launcher for the repair tool.
java/tools/README.md	Documents the new repair tool in the tools README.
java/tools/README-zh.md	Adds Chinese documentation for the new repair tool.
java/common/src/main/resources/org/apache/tsfile/i18n/messages.properties	Adds i18n messages for tool usage/errors/output.
java/common/src/main/resources/org/apache/tsfile/i18n/messages_zh.properties	Adds Chinese i18n messages for tool usage/errors/output.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

jt2594838 · 2026-07-30T09:33:36Z

+    tablePointCounts.forEach(
+        (tableName, pointCount) ->
+            metadata.addProperty(
+                TsFileIOWriter.TABLE_POINT_COUNT_PROPERTY_PREFIX + tableName,
+                Long.toString(pointCount)));


This is intentionally overwritten by design. The user should not specify this or use our key.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

jt2594838 added 4 commits July 29, 2026 12:17

add property read-write interface

91734b3

May record table point count

395e57a

Add perf test

9524765

Add table point count repair tool

5ffa041

jt2594838 added 2 commits July 30, 2026 12:32

add point-count tool

efed191

add point-count tool

9d097aa

jt2594838 commented Jul 30, 2026

View reviewed changes

ColinLeeo requested a review from Copilot July 30, 2026 09:12

Copilot started reviewing on behalf of ColinLeeo July 30, 2026 09:23 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

jt2594838 and others added 4 commits July 30, 2026 17:34

supplement usage for other os

d327fae

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

supplement usage for other os

339a1f1

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

use byte[] for properties

888166d

use byte[8] for table point count

a095b07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support TsFile properties and optional table point counting - #882

Support TsFile properties and optional table point counting#882
jt2594838 wants to merge 10 commits into
developfrom
add_property_read_write_interface

jt2594838 commented Jul 29, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 29, 2026 •

edited

Loading

Uh oh!

jt2594838 Jul 30, 2026

Uh oh!

jt2594838 Jul 30, 2026

Uh oh!

jt2594838 Jul 30, 2026

Uh oh!

jt2594838 Jul 30, 2026

Uh oh!

jt2594838 Jul 30, 2026

Uh oh!

jt2594838 Jul 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

jt2594838 Jul 30, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jt2594838 commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Point-count semantics

Table point-count repair tool

Verification

Uh oh!

codecov-commenter commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jt2594838 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

jt2594838 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

jt2594838 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

jt2594838 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

jt2594838 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

jt2594838 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

jt2594838 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jt2594838 commented Jul 29, 2026 •

edited

Loading

codecov-commenter commented Jul 29, 2026 •

edited

Loading