Skip to content

[python][ray] Incrementally commit update_by_row_id file groups#8826

Draft
XiaoHongbo-Hope wants to merge 1 commit into
apache:masterfrom
XiaoHongbo-Hope:update_by_row_id_incremental_commit
Draft

[python][ray] Incrementally commit update_by_row_id file groups#8826
XiaoHongbo-Hope wants to merge 1 commit into
apache:masterfrom
XiaoHongbo-Hope:update_by_row_id_incremental_commit

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

  • Add an opt-in max_groups_per_commit parameter to update_by_row_id.
  • Deliver completed _FIRST_ROW_ID file groups to the driver as Ray workers finish, and commit each bounded window through a reusable StreamTableCommit.
  • Keep the existing single atomic commit behavior when the option is unset.
  • Preserve the original base snapshot on every commit message so conflict detection still covers the full read-to-commit window.
  • Document that incremental mode may leave earlier windows committed if a later window fails.

Tests

  • pytest -q pypaimon/tests/ray_update_by_row_id_test.py pypaimon/tests/table_merge_into_test.py pypaimon/tests/table_update_test.py (138 passed, 16 subtests passed)
  • Multi-worker Ray coverage with 8 input blocks, 4 shuffle partitions, 8 file groups, and 3 groups per commit.
  • flake8 --config=dev/cfg.ini on changed Python files.
  • Python 3.6 py_compile compatibility check.
  • git diff --check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant