Repository navigation
feat(array): min/max for fixed-size lists, so UUID columns get zone-map stats - #10457
XiangpengHao wants to merge 3 commits into
Conversation
…ap stats `vortex.uuid` is stored as `FixedSizeList(U8, 16)`, and the min/max aggregate skipped fixed-size lists as a stop-gap (vortex-data#8243), so UUID zone maps and footers kept only null counts and `uuid = x` pruned nothing. Compute min/max for fixed-size lists of fixed-width elements (bool, primitive, decimal), ordered by the compare kernel's own row comparator so statistics and comparisons always agree. Non-nullable byte lists such as UUIDs compare as byte slices. Lists wider than 64 bytes per value get none, so embedding-style columns do not grow zone maps. DuckDB's COPY RETURN_STATS and the JNI writer now convert UUID and fixed-size-list bounds instead of failing on them. Signed-off-by: XiangpengHao <xiangpeng@spiraldb.com>
Signed-off-by: XiangpengHao <xiangpeng@spiraldb.com>
Merging this PR will degrade performance by 24.52%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | decode_primitives[i64, (1000, 32)] |
24.7 µs | 38 µs | -35.15% |
| ❌ | WallTime | bitpack_blocked_compress_avx2 |
6.7 µs | 7.6 µs | -12.14% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing XiangpengHao:xiangpeng/fsl-min-max (5b9c1d0) with develop (fdba755)
Summary
vortex.uuidis stored asFixedSizeList(U8, 16), and the min/max aggregate skips fixed-size lists as a stop-gap (#8243, for #8189). UUID zone maps and file footers therefore keep only null counts, anduuid_col = xprunes nothing, which hurts sorted UUIDv7 keys most. This implements min/max for fixed-size lists, so UUID columns get the same zone-map and footer statistics, and the same pruning, as Parquet's UUID logical type.Changes
FixedSizeList(aggregate_fn/fns/min_max/fixed_size_list.rs): lists of fixed-width elements (bool, primitive, decimal), ordered lexicographically by the compare kernel's own row comparator (compare/nested.rs), so statistics and comparisons always agree. Null lists are skipped, null elements order first, floats use total order. Non-nullableu8lists (the UUID case) compare as byte slices, the same order as Postgres, DuckDB and Parquet.MAX_FIXED_SIZE_LIST_MIN_MAX_BYTES(64 bytes per value) get no min/max, so embedding-style columns don't grow zone maps for pruning they can't use. The result is the same for every encoding.vortex.uuidgets min/max through its storage; no UUID-specific code.COPY … RETURN_STATSconverts UUID bounds toUUIDand fixed-size-list bounds toARRAYinstead of failing on them; the JNI writer converts them tojava.util.UUIDandList.vortex/tests/uuid_pruning.rs(a UUID column over four zones;uuid = xkeeps only the zone that can hold it, footer min/max,can_prune),fixed_size_list::tests(ordering vsScalar::partial_cmpandcompare, nulls, floats, sliced and dictionary inputs, the cap at its boundary),vortex-uuidmin/max order, DuckDB conversion tests andcopy_return_stats.slt.JNIWriterTest.testUuidAndFixedSizeListBoundsis added but was not run locally (no JDK).API Changes
Files written with this change carry min/max for UUID and narrow fixed-size-list columns. Engines that receive file statistics (DuckDB
RETURN_STATS, the JNI writer's column statistics) now see bounds for those columns.