Skip to content

[python] Use Native StreamTableScan for streaming reads - #10434

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-python-parity-20261008-followup
Oct 8, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-python-parity-20261008-followup

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Use Rust's stateful StreamTableScan for Native streaming reads instead of calling a public SnapshotReader once for each Python-selected frame.

  • Rust owns initial/follow-up selection, delta/changelog mode, cursor, watermark, checkpoint restoration, bucket filtering and consumer persistence.
  • Python only converts splits, polls when no plan is available, and acknowledges a checkpoint after the caller resumes following a yielded plan.
  • Empty follow-up plans advance acknowledged progress without waiting for a yielded frame; failed notifications retain pending checkpoints for retry.
  • Initial full reads and follow-ups preserve canonical read_type, nested/map projection, row kinds and opaque Native splits.
  • Native planning errors propagate without switching planners mid-stream. Optional runtime/catalog initialization can still select the Python backend; query-authorized scans retain their Python authorization path.
  • Replace tests using private per-snapshot helpers with public stream checkpoint/restore/plan calls. Update Native coverage tracking and enable the existing nested streaming regression against the current read-type API.

Depends on apache/paimon-rust#1075 reaching Rust main. Native CI continues to build main; this PR stays draft while that dependency is outstanding.

Validation

  • 21 read/write/update/merge/schema/consumer/Blob suites with all five Native flags: 1,321 passed, 42 subtests passed.
  • Native plans 1,182; reads 1,435; writes 1,302; commits 45; update paths exercised.
  • Python binding stream tests: 12 passed.
  • Flake8, Python 3.6 syntax parsing of changed Python files, and diff checks: passed.
  • Independent verification covered failed plan/notification/conversion retries and consumer acknowledgement timing.

@JingsongLi JingsongLi changed the title [python] Use Native SnapshotReader for streaming frames [python] Use Native StreamTableScan for streaming reads Oct 8, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 4d887ba4b641656b26575ba00c5f5be714049986 together with Rust dependency PR #1075.

I found no blocking issues. The Python adapter preserves the configured canonical read type, converts native plans before acknowledging progress, retries conversion and notification failures without losing the pending checkpoint, and keeps the existing Python fallback/authorization boundaries. The Rust dependency has been validated at its current head; the Python CI checks are still running.

@JingsongLi
JingsongLi marked this pull request as ready for review October 8, 2026 11:20
@JingsongLi
JingsongLi merged commit 487a411 into apache:master Oct 8, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants