scalar_at receives only an ArrayView, so it doesn't cache anything — validity, child wrappers and any decode are rebuilt on every row. probe_scalar receives a ProbeState, which carries the encoding's own retained state plus a child probe per slot, so a repeated read keeps its preparation.
Example migration: #9905
Writing a probe_scalar
- Cache in
type ProbeState anything the encoding owns itself — a decoded page, a rebuilt array, a cursor. Use () when there's nothing.
- Don't cache anything owned by a child. Read children through
state.slot(..), which hands back that child's own probe and so uses the child's cache.
- Leave
scalar_at as Self::probe_scalar(&mut ProbeState::once(array), index, ctx) until the vtable method is removed.
Encodings to migrate
(Primitive and Struct are done; Slice, Dict, Chunked, Shared, Masked, Extension, FoR, ZigZag, ALP, FSST, DateTimeParts are in #9905.)
Steps
- Migrate each encoding above, adding it to
test_repeated_probe_consistency in the conformance suite.
- Make
probe_scalar a required method; deprecate, then remove, OperationsVTable::scalar_at.
- Over the next few releases, remove
ArrayRef::scalar_at (already deprecated) and deprecate then remove ArrayRef::is_valid / is_invalid in favour of probe() / repeated_probe().
scalar_atreceives only anArrayView, so it doesn't cache anything — validity, child wrappers and any decode are rebuilt on every row.probe_scalarreceives aProbeState, which carries the encoding's own retained state plus a child probe per slot, so a repeated read keeps its preparation.Example migration: #9905
Writing a
probe_scalartype ProbeStateanything the encoding owns itself — a decoded page, a rebuilt array, a cursor. Use()when there's nothing.state.slot(..), which hands back that child's own probe and so uses the child's cache.scalar_atasSelf::probe_scalar(&mut ProbeState::once(array), index, ctx)until the vtable method is removed.Encodings to migrate
BoolConstantDecimalFilterFixedSizeListInterleaveListListViewMapNullPatchedPiecewiseSequenceScalarFnUnionVarBinVarBinViewVariantALPRDBitPackedByteBoolDecimalBytePartsDeltaOnPairParquetVariantPcoRLERunEndSequenceSparseTransposedBoolZstdZstdBuffersPythonVTableNotSupported(
PrimitiveandStructare done;Slice,Dict,Chunked,Shared,Masked,Extension,FoR,ZigZag,ALP,FSST,DateTimePartsare in #9905.)Steps
test_repeated_probe_consistencyin the conformance suite.probe_scalara required method; deprecate, then remove,OperationsVTable::scalar_at.ArrayRef::scalar_at(already deprecated) and deprecate then removeArrayRef::is_valid/is_invalidin favour ofprobe()/repeated_probe().