Skip to content

[mosaic] Expose column-batch reads and writer extension hooks - #10516

Merged
JingsongLi merged 1 commit into
apache:masterfrom
jianguotian:feat/mosaic-column-batch-hooks-upstream-20261012
Oct 12, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
jianguotian:feat/mosaic-column-batch-hooks-upstream-20261012

Conversation

@jianguotian

Copy link
Copy Markdown
Contributor

Purpose

Mosaic reads Arrow batches internally, but currently exposes row iterators to callers. Expose those batches through Paimon's existing vectorized reader contract so downstream engines can process columns without unpacking rows.

  • ArrowBatchReader and MosaicRecordsReader expose independent batch wrappers and release Arrow buffers per consumed batch, preserving projection, prefetch and row-tracking fallback.
  • MosaicRecordsWriter exposes its conversion buffer, flush and failed-state check to subclasses. Failed native writes reject further input and are not retried during close.
  • MosaicWriterFactory adds an overridable creation hook while retaining option validation.

Existing bundle interfaces, column-to-Arrow conversion and native Mosaic writing are reused. MosaicRecordsWriter continues to use the row fallback for ordinary VectorizedBundleRecords; consumers can implement batch-write adapters through the new hooks.

Tests

Passed clean Maven verification on JDK 8 (linux/aarch64): 131 tests, zero failures, errors or skips. Checkstyle, Spotless and Enforcer checks were enabled.

mvn -B -pl paimon-arrow,paimon-mosaic -am \
  -Dtest=ArrowBatchReaderTest,Mosaic*Test -DwildcardSuites=none \
  -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false clean verify

Coverage includes prefetch, projection, row tracking across row groups, deletion vectors, batch lifetime, native-write failures, external writer subclass round trips and factory option validation.

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 7246bdc into apache:master Oct 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants