feat(scan): read bucket metadata for manifest pruning

[jianguotian]

Purpose

Use bucket metadata written by Java Paimon to skip irrelevant manifest files before Rust fetches and decodes their contents.

This PR is read-side only. It does not add or modify any Rust manifest-writing, compaction, or sorting behavior.

Changes

• Decode the Java-compatible nullable _TOTAL_BUCKETS field from external manifest-list writer schemas without adding it to Rust's manifest-list writer schema.
• Before fetching manifest contents, combine _MIN_BUCKET, _MAX_BUCKET, and _TOTAL_BUCKETS with the existing bucket-key predicate hash calculation.
• Apply the same conservative pruning to full snapshot and incremental manifest-list scans.
• Fail open for legacy, missing, invalid, mixed, or unassigned-bucket metadata.
• Retain existing entry-level filtering as a correctness backstop.
• Add a regression test proving Rust reads _TOTAL_BUCKETS but does not serialize it.

Scope

• no TableCommit changes
• no Rust emission of _TOTAL_BUCKETS
• no manifest sorting or compaction changes
• no dependency on the closed write-side sorting PRs

Verification

• cargo fmt --check
• cargo test -p paimon --lib — 2,835 passed, 2 ignored
• cargo clippy -p paimon --lib --tests -- -D warnings -A clippy::unnecessary-cast
  • the allowance is for an unrelated existing Rust 1.95 lint in lumina/ffi.rs
• dedicated coverage for external _TOTAL_BUCKETS decoding, exact bucket selection, legacy fail-open behavior, unassigned buckets, and invalid bucket envelopes

Related Java work

• [core] Support bucket pruning for manifest files paimon#9791

[JingsongLi]

Requirement fit: SUPPORTED. Implementation: CLEAN.

The new read-only _TOTAL_BUCKETS field is decoded from external manifest schemas without changing Rust serialization. Manifest filtering only acts when bucket metadata is internally valid and otherwise falls back to the existing scan, while retaining entry-level filtering as a correctness backstop. I found no regression in the changed paths. Verification: the dedicated external-schema and bucket-pruning tests passed. Review scope: metadata compatibility, bucket envelopes, legacy/invalid metadata, and full/incremental scan call sites.

[JingsongLi]

+1