Repository navigation
feat(python): expose core vector search and scored results - #1087
Conversation
leaves12138
left a comment
leaves12138
left a comment
There was a problem hiding this comment.
Reviewed at cd2e2d1f14e83893513c4b487e432b1da04ad0f1 paired with apache/paimon#10476 at c846e956d4751d58b08673ff4b9a690ec53787fc. Requesting changes for Java DE scalar-filter parity and the public Python keyword contract.
Independent validation on the fixed heads:
- Rebuilt and installed the companion Native extension from the reviewed Rust head.
- Full Rust core unit suite: 3877 passed, 6 ignored.
- Python binding vector/read/table/resolved-table suites: 197 passed.
- New real-REST Native vector suite: 30 passed.
- Fourteen related PyPaimon suites with all five Native switches enabled: 997 passed and 96 subtests passed; every update-family counter was nonzero.
- Formatting, changed-file Flake8, and Python 3.6 syntax checks passed.
- Additional reviewer Java/API parity cases: 13 failed, 6 passed. These expose the two DE scalar-filter semantic differences plus the single-vector keyword mismatch described inline.
The passing baseline suites do not cover these parity cases. Java references are the companion head's AbstractDataEvolutionVectorRead.prepareSplits (lines 159-181) and scalarMatchedRows (lines 246-266). The reproduced reference outputs use the classic Python path, checked against those Java implementations; I did not run a separate Java process.
| builder.with_query_vector(query.clone()); | ||
| } | ||
| let results = py | ||
| .detach(|| runtime().block_on(builder.execute())) |
There was a problem hiding this comment.
[P1] Preserve Java's raw-search routing for unevaluable scalar filters
This exposes the core DE filtered-search path, but DeVectorScan::plan always resolves the predicate by reading matching row IDs and passes that allow-list to ANN. That is not Java's AbstractDataEvolutionVectorRead.prepareSplits behavior: when no scalar index can evaluate the predicate, Java moves the covered vector-index ranges to raw scoring and applies the residual there. The companion Python PR implements that routing in its classic reader, then bypasses it here through Native execution.
I reproduced the difference on a real REST table using the companion vector_table fixture, 50 near vectors and 50 distant vectors in the same partition, an IVF-flat index with nlist=4, query [0, 0], filter id >= 100 with no scalar index on id, ivf.nprobe=1, and limit 3. The classic/reference path returns 3 matching hits; this binding and the new PyPaimon Native path both return 0. The difference occurs for single and batch searches in FAST, FULL, and DETAIL. Setting Native nprobe to 4 restores all 3 hits, confirming that filtered hits are lost through ANN candidate selection rather than being absent from the data.
Please implement Java's scalar-evaluation/range-routing distinction in core before exposing this path, including both single and batch readers. A data-derived allow-list sent to ANN is not a substitute for the required raw fallback.
| builder.with_options(options.clone()); | ||
| } | ||
| for filter in &self.filters { | ||
| builder.with_filter(filter.clone()); |
There was a problem hiding this comment.
[P2] Honor the Java scalar-candidate refinement switch
Both the single and batch configuration paths delegate all filters to core's unconditional matching_row_ids_for_filter data read. For an inexact scalar index answer this ignores global-index.filter.refine-from-data=false. Java's AbstractDataEvolutionVectorRead.scalarMatchedRows explicitly excludes those candidates when refinement is disabled; it does not read the data to turn them into exact matches.
A real REST reproduction uses names alpha, beta zeta, and gamma, an IVF-flat index on the vector column, a BTREE index on name, filter name LIKE '%zeta%', and limit 1. With refinement disabled the classic/Java-reference policy returns 0 hits, but both the Rust binding and Native PyPaimon return 1; batch returns [1, 1] instead of [0, 0]. All three search modes reproduce it. With refinement enabled the paths agree. Please preserve exact/unevaluable/inexact scalar evaluation state and apply the configured refinement policy before ANN, rather than always materializing an exact row-ID filter.
| slf.config.limit = Some(limit); | ||
| slf | ||
| } | ||
| fn with_query_vector(mut slf: PyRefMut<'_, Self>, vectors: Vec<f32>) -> PyRefMut<'_, Self> { |
There was a problem hiding this comment.
[P2] Match the published single-vector keyword argument
The new stub declares with_query_vector(self, vector: Sequence[float]), and the existing PyPaimon builder accepts that same vector keyword, but this PyO3 method names the argument vectors. Consequently builder.with_query_vector(vector=[1.0, 0.0]) raises TypeError: ... got an unexpected keyword argument 'vector' despite matching the public type signature. Positional-only test calls miss this. Please rename the parameter to vector (or set the equivalent explicit Python signature) and add a keyword-call regression.
leaves12138
left a comment
leaves12138
left a comment
There was a problem hiding this comment.
Re-reviewed at 2a2692f5b85ab9317f4257982b37a7af3945ab6a, paired with apache/paimon#10476 at 7e6453d1a8ba8a80da647bc5e520260061248006. All three previous findings are fixed: the 19 independent regression cases now pass, including unevaluable-filter raw routing, the candidate-refinement switch, and the vector= keyword.
One Java-parity blocker remains: scalar-unindexed ranges are not added to raw vector scoring when scalar coverage is older/smaller than vector coverage and scalar-index.search-mode=full/detail. This silently omits matching rows on both the direct binding and the public Native PyPaimon path. See the inline reproduction; this is the Java coverage contract, not an additional requirement inferred only from Python behavior.
Independent validation on these fixed heads:
- Rebuilt and installed the Native extension from this Rust head.
- Full Rust core unit suite: 3882 passed, 6 ignored.
- Binding vector/read/table/resolved-table suites: 198 passed.
- Companion real-REST Native vector suite: 54 passed.
- Fourteen related PyPaimon suites with all five Native switches enabled: 997 passed, 96 subtests passed; all six update-family counters were nonzero.
- Previous reviewer parity suite: 19 passed.
- Additional coverage matrix: 12 failed, 6 passed (scalar FULL/DETAIL fail for single/batch across vector FAST/FULL/DETAIL; scalar FAST controls pass).
- Rust formatting and changed Python files' Python 3.6 syntax checks passed. The companion Python PR has two new configured Flake8 E501 errors, noted there.
Java semantics were checked against the companion fixed head's DataEvolutionVectorScan and DataEvolutionGlobalIndexCoverage; the executable reference in the coverage matrix is classic PyPaimon on the same real REST tables. No separate Java process was run.
| } | ||
|
|
||
| if search_mode != GlobalIndexSearchMode::Fast { | ||
| if route_indexed_to_raw || search_mode != GlobalIndexSearchMode::Fast { |
There was a problem hiding this comment.
[P1] Include scalar-unindexed ranges in raw vector scoring
This raw fallback is still entered only for an unevaluable scalar predicate or a non-FAST vector mode, and its ranges are calculated only from vector coverage. An evaluable but partially covered scalar index returns Some(empty) for a matching row outside its coverage, so route_indexed_to_raw stays false. If the vector index already covers that row, FULL/DETAIL also have no vector-unindexed range to recover it.
Real REST reproduction using the companion vector_table fixture: (1) create a BTREE index on id over the initial six rows; (2) append {id: 7, embedding: [3, 3], pt: 1}; (3) create an IVF-flat index on embedding with dimension 2 and nlist 1, now covering all seven rows; (4) search [0, 0] with id = 7, limit 10, and scalar-index.search-mode=full or detail. Classic returns 1 hit; both the direct Rust binding and Native PyPaimon return 0 (batch: [1, 1] versus [0, 0]). All three vector modes reproduce this. Scalar FAST controls correctly return no hit on both paths. Python search was patched to raise during the Native calls, so fallback cannot mask the reproduction.
Java explicitly adds scalar-unindexed ranges using the configured scalar search mode in DataEvolutionVectorScan lines 174-191, intersecting them with vector coverage when vector mode is FAST. Please preserve that rule here: compute predicate-field scalar coverage, union its permitted uncovered ranges into raw scoring even when scalar evaluation succeeds, and apply the residual predicate there. In FAST vector mode, restrict only these scalar fallback ranges to vector-covered rows.
leaves12138
left a comment
leaves12138
left a comment
There was a problem hiding this comment.
Re-reviewed at 68da723a4014508bb6584992bb3d4d0819d7e86d, paired with apache/paimon#10476 at 4e87ecdebab3c6dd9f72444ade0394655dfd24cf. The remaining scalar-coverage finding is fixed. No new actionable blocker found in the reviewed changes.
The raw fallback now includes scalar-unindexed ranges according to the effective scalar search mode, intersects those ranges with vector coverage for vector FAST, merges overlaps, and applies the residual predicate before Top-K. This matches the coverage/routing contract in Java DataEvolutionVectorScan lines 174-191 and DataEvolutionGlobalIndexCoverage. Query-level family modes also override table modes. The earlier raw-routing, refinement-switch, and vector= keyword fixes remain covered.
Independent validation on these exact heads, using a rebuilt and installed Native extension:
- Full Rust core unit suite: 3884 passed, 6 ignored.
- Binding vector/read/table/resolved-table suites: 198 passed.
- Real-REST Native vector module: 94 passed, with 49 Native plans, 51 Native reads, 158 Native writes, and 295 Native commits exercised.
- Reviewer parity cases: 49 passed (37 prior-finding/coverage cases plus 12 additional query-mode override cases). The direct binding and public Native paths were checked against classic results; Python single/batch readers were patched to raise during Native calls. This includes query FAST overriding a FULL table without mutating table options, with either index family being older.
- Fourteen related Native suites: 997 passed, 96 subtests passed; all six update-family counters were nonzero.
- Classic vector-filter suite: 79 passed, 19 subtests passed.
- Rust fmt, configured changed-file Flake8, and Python 3.6 syntax checks passed; the companion's two previous E501 findings are fixed.
Java was used as the source-level semantic reference; no separate Java process was run. These are local/real-REST correctness checks, not production-storage or performance validation.
2 participants
Summary
Expose the existing Rust core vector-search pipeline to Python so PyPaimon can execute local searches without coordinating index evaluation, scalar filtering, raw scoring and refinement in Python.
Java references
VectorSearchBuilderImpl,BatchVectorSearchBuilderImpl,AbstractDataEvolutionVectorRead,PrimaryKeyScoredResult, andIndexedSplitare the semantic references. Core already supports both DE and PK searches; this adds the Python surface and closes the gaps found during integration.Initial verification
cargo test --locked -p paimon --lib: 3876 passed, 6 ignored (including 341 vector tests).cargo test --locked -p paimon --lib table::source::tests: 57 passed.cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warningsand pinned formatting check passed.Companion PyPaimon PR: apache/paimon#10476. Its full Native regression with plan/read/write/update/commit enabled passed 3000 tests and 127 subtests locally; 6 tests were skipped. It remains Draft until this API is available on Rust main.
Review fixes
global-index.filter.refine-from-data=trueand otherwise excluded; unevaluable filters route vector-index-covered ranges to raw scoring in FAST/FULL/DETAIL.with_query_vector(vector=...)with the published Python signature.Verification for commit
2a2692f:1.98.0,fulltext,vortex,-D warnings) passed.Scalar coverage review repair
Commit
68da723fixes the remaining Java coverage finding. Scalar FULL/DETAIL now contribute predicate-field coverage gaps to raw scoring even when the scalar index lookup succeeds with an empty match set. Vector FAST intersects those extra ranges with the vector-index corpus; vector FULL/DETAIL retain their own uncovered ranges. Raw scoring executes the residual predicate against the pinned snapshot.Search modes use table options overlaid by query options, matching
DataEvolutionVectorScan.effectiveCoreOptions(). Coverage accounting reuses the existing range utilities, intersects coverage across all predicate fields, and excludes composite and other index families from scalar coverage.Current repair verification:
1.98.0,fulltext,vortex,-D warnings) passed.