Skip to content

[Core] Preserve captured snapshot metadata and empty read views - #1098

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-pinned-read-views
Oct 11, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-pinned-read-views

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

What changes

  • Let Table::copy_with_pinned_snapshot capture complete snapshot metadata or an explicitly empty read view, preserving the caller-resolved read schema.
  • Reuse that state in snapshot resolution, vector prepared filters, manifest/index/partition metadata and DataFusion partition count execution.
  • Normalize the supported point/incremental selectors when pinning; invalidate changed read selections and prevent timestamp windows from bypassing the stale-schema guard.
  • Plan primary-key vector data and index files from one complete snapshot, without reopening expired snapshot JSON. Validate the original scan configuration first and follow Java's rejection of nonempty incremental PK vector plans.
  • Add a thin Python binding to pass Java-format Snapshot JSON or None into the core Table.

Why

PyPaimon already captures a snapshot before multimodal search and lookup. Its Native bridge currently transports only schema/options, so a captured empty view follows later commits and retained-tag searches can reopen an expired snapshot file. The existing Rust cache cannot distinguish an unresolved view from an explicitly empty one.

The implementation reuses the existing Table cache and resolution path. Snapshot metadata is shared across table copies. Java PrimaryKeyVectorScan and search read plans provide the reference for reusing one complete snapshot across data/index/read stages.

Tests

  • Before the fix: all 8 REST empty-view/retained-tag Native regression cases failed.
  • 260 passing Rust tests across snapshot, PK vector, DE vector, batch vector, hybrid, full-text and MERGE suites.
  • 16 passing DataFusion tests: metadata empty-view regression and all partition count pushdown tests.
  • 154 passing Python tests with all five Native CI flags, including captured read views, hybrid, the entire Native MERGE suite and update/write/commit smoke tests. Also 110 passing existing Native vector/search-snapshot tests. The wider search run passed its full-text cases and caught an internal MERGE scan-mode rewrite, fixed by using the common pinning method and verified by the MERGE reruns.
  • Rust format and Clippy on 1.98.0; Python binding build.

Paired Python PR: apache/paimon#10497 (draft until this Rust API is merged). It enables these existing Python read views through the common Native Table bridge and keeps CI building upstream Rust main.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed Rust PR #1098 at head 5eb8cd6. The captured snapshot/empty-view handling is consistent across core scans, hybrid/PK/vector search, DataFusion system tables, and Python bindings. Targeted validation passed: cargo fmt --check; hybrid core tests 20/20; DataFusion hybrid tests 4/4; partition-count empty-view test 1/1; system-table empty-view test 1/1; time-travel tests 29/29; empty vector-filter and PK-vector retained-tag tests passed. No blocking issues found.

@JingsongLi
JingsongLi merged commit 44cda80 into apache:main Oct 11, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants