Add planner-owned selection and bounded ListLayout scans to V2 by joseph-isaacs · Pull Request #10537 · vortex-data/vortex

joseph-isaacs · 2026-10-16T09:26:20Z

Masked scans apply selection through a planner before projection. ListLayout scans resolve selected offsets and validity before reading elements, and the projection planner can subdivide large fixed-width element ranges into independently scheduled work.

Based on develop 92a7db079. The default implementation owns subdivision, a shared 32 MiB decoded-work admission budget, child execution and ordered result collection. Small, repeated-segment, oversized-segment and unsupported work retain the serial pipeline.

PlannerOutput::Await lets the driver suspend a planner while its child tasks finish; existing Planner, Morsel, Source, IoSource and SegmentSource methods are unchanged.
Selection-aware lazy lowering, bounded dense-list grouping and chunk-buffer reuse limit work on unselected rows.
Regression coverage includes outstanding IO during/after a planner await, cancellation, serial fallback, retired pipelines and nullable fieldless structs.
Standard random access uses the V2 scan builder for both local and object-store Vortex reads.

The opt-in comparison path (VORTEX_LIST_PIPELINE=1) now hands selected element views to the enclosing planner. List sources first resolve offsets and validity, then describe the element plan with selected row ranges and masks. The planner builds a separate small graph for that view and returns its arrays to the waiting list source.

SelectedPlan holds a shared physical plan and selected splits; constructing it performs no compilation.
The source waits through the existing Blocked(Child) protocol. SourceReady::Activate, Source::activate, selection gates, and mutation of a running graph are removed.
Nested lists recursively hand selected views to the planner. Eligible fixed-width work retains the existing parallel subdivision and 32 MiB admission budget. Other views run without holding that budget.
Results retain their selected split index so IO completing out of order cannot reorder list elements.
Added regressions cover pruning unselected unsupported children, empty selections, shared reads, nullable lists, selected-split ordering, cancellation, and IO failure.

The comparison flag preserves the existing default path for matched off/on benchmarks. It has not been enabled by default.

Local microbenchmarks ran sequentially on a 14-logical-CPU Apple Silicon Mac with release_debug. Each query selects 100 lists; the benchmark excludes disk IO and Arrow conversion. Two alternating off/on rounds used 100 samples of 100 iterations in the same binary. Values are medians of round medians:

Selection	Chunks	Current µs	Selected view µs	Change
Contiguous	8	3.510	3.420	-2.6%
Contiguous	128	3.382	3.452	+2.1%
Contiguous	1024	4.286	4.325	+0.9%
Uniform	8	47.160	46.525	-1.3%
Uniform	128	84.185	76.235	-9.4%
Uniform	1024	87.065	79.375	-8.8%

The six-fixture full-Arrow comparison passed all 72 reference checks and completed 32 timing cases (three seeds, two alternating rounds, 64 queries per run, first eight discarded). Most cases were within 4%. Two regressions persisted in a follow-up with three alternating rounds, three seeds, and 128 queries per run: cached short clustered lists with 64 KiB chunks measured 0.178 → 0.193 ms (+8.1%); a large nested-list singleton with 64 KiB chunks and reopen measured 0.452 → 0.490 ms (+8.5%). Large nested uniform selections with 1 MiB chunks measured 11.063 → 11.190 ms. The earlier activation implementation's speedup on that case is not retained. Small-list uniform selections with 64 KiB chunks measured 0.918 → 0.919 ms.

Random-access benchmarking uses the existing standard suite. The custom ListLayout fixture generator, comparison script, scan example, and extra CI steps have been removed. The workflow has no changes relative to the PR base. Adding action/bench-random-access runs the original suite.

A local branch comparison with the ListLayout override and VORTEX_LIST_PIPELINE disabled used identical standard nested-list data, Rust 1.98.0, release_debug, three alternating rounds, and five seconds per target. PR 1107f53c34 versus develop 8c2534ea18:

Access	Develop	PR	Change
Clustered, cached	223 µs	194 µs	-12.9%
Clustered, reopen	392 µs	338 µs	-13.8%
Uniform, cached	1.900 ms	1.810 ms	-4.7%
Uniform, reopen	2.096 ms	2.005 ms	-4.3%

Develop's dependency updates are included in this branch comparison. The cleanup commit changes no benchmark runtime code. The complete local Vortex matrix finished: all four datasets, both patterns and open modes, plus legacy taxi cases (18 cases, three alternating rounds, three seconds per target). The same release binaries were used; the cleanup commit only removes benchmark helpers and CI steps. Positive percentages below are slower than develop.

Dataset	Clustered cached	Clustered reopen	Uniform cached	Uniform reopen
Taxi	+8.0%	+7.9%	-0.4%	+2.2%
Feature vectors	-13.7%	-8.8%	-13.3%	-6.3%
Nested lists	-12.3%	-13.8%	-2.8%	-6.6%
Nested structs	+11.6%	+8.2%	-1.4%	+0.2%

Legacy taxi improved 6.9% cached and 9.8% with reopen. Clustered taxi regressed about 8%, and clustered nested structs regressed 8–12%. These repeated across the alternating rounds, so this PR is not regression-free. Other-format baselines and remote/S3 performance were not measured locally in this comparison; the requested standard CI suite includes the other formats.

Release benchmark builds and the measurements above completed. New unit regressions, linting, and formatting checks were not run locally.

Prior validation of the default implementation at c4fa7818b: 692 layout and 79 scan library tests, all seven await-protocol tests, scoped all-target/all-feature clippy with -D warnings, and 200 real-file reference comparisons plus five malformed-selection checks.

The default planner retains a previously measured approximately 9–10% regression versus the rebased pre-refactor control for a large nested-list singleton with 1 MiB chunks (0.241 → 0.264 ms on the earlier ARM host). IO measurements attributed that gap to batching and reduced read overlap. Its 32 MiB estimate covers in-flight decoded work, not retained output or coalesced encoded buffers; the full memory matrix has not been revalidated.

This remains a draft. Standard benchmark CI for fe25a60931 is pending.

codspeed · 2026-10-16T09:34:33Z

Merging this PR will degrade performance by 27.68%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

bench_compare_primitive[(10000, 2)]

⚠️

23 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

list_scan[false, 8]

random_i128[0.01]

test_scan_to_arrow_array[int_float]

test_repeated_scan_scalar_at[int]

test_scan_filter[int_float]

test_repeated_scan_filter[int_float]

test_scan_scalar_at[int]

test_repeated_scan_scalar_at[int_float]

test_scan_filter[int_str]

test_scan_filter[int]

…and 13 more

ℹ️ 40 benchmarks have no flame graph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

test_scan_to_arrow_array[int_float]

test_repeated_scan_scalar_at[int_str_float]

test_repeated_scan_scalar_at[int]

test_scan_filter[int_float]

test_scan_scalar_at[int_float]

test_repeated_scan_filter[int_float]

test_scan_scalar_at[int]

test_repeated_scan_scalar_at[int_str]

test_repeated_scan_scalar_at[int_float]

test_repeated_scan[int]

…and 30 more

❌ 7 regressed benchmarks
✅ 2401 untouched benchmarks
🆕 6 new benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	WallTime	`filter_neon[64]`	1.8 µs	3.4 µs	-47.05%
❌	WallTime	`filter_neon[8]`	959 ns	1,620 ns	-40.8%
❌	WallTime	`threshold_neon[80]`	3.5 µs	4.9 µs	-28.21%
❌	WallTime	`threshold_neon[8]`	1.4 µs	1.9 µs	-25.45%
❌	WallTime	`filter_neon[80]`	2.7 µs	3.3 µs	-19.25%
❌	WallTime	`threshold_neon[64]`	2.7 µs	3.1 µs	-13.74%
❌	Simulation	`isolated_runs[u32, (16, 0)]`	78.5 µs	88.6 µs	-11.44%
🆕	Simulation	`list_scan[false, 1024]`	N/A	254.9 µs	N/A
🆕	Simulation	`list_scan[false, 128]`	N/A	238 µs	N/A
🆕 ⚠️	Simulation	`list_scan[false, 8]`	N/A	591.7 µs	N/A
🆕	Simulation	`list_scan[true, 1024]`	N/A	1.9 ms	N/A
🆕	Simulation	`list_scan[true, 128]`	N/A	1.9 ms	N/A
🆕	Simulation	`list_scan[true, 8]`	N/A	1.3 ms	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing ji/decompression-v2-scan (fe25a60) with develop (8c2534e)}

12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Run the unchanged full 48-case Arrow comparison and 36-case standard suite, including parent and candidate V1 controls for the standard suite. Use the same CI-only wrapper as ccbe15b and baseline 5231b70. Runtime source is 8442d46, tree-equivalent to local 95fda7fc8. The previous run 38972129118 is complete. It confirmed the offset-copy feature-vector improvements but retained one +5.2% Arrow rerun. This run checks the combined offset-copy, generated-offset-statistics, and guarded list-overlap changes. Keep PR #10537 draft; this run is not a merge gate. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Rebase draft PR #10537 onto develop 92a7db0. Apply selections through a composable planner, resolve list offsets before selected elements, and split substantial fixed-width child projections at stored segment boundaries. The projection planner owns subdivision, byte-budget admission, child task execution and ordered collection. Unsupported splits retain the serial path. Keep Planner and Morsel trait methods unchanged. Replace the experimental completion callbacks, WorkFuture and ChildExecutor traits with one PlannerOutput::Await variant; morsels only publish the assembled result. Preserve develop's global local-file coalescing configuration. The 8 KiB experiment caused tiny-lookup regressions and is not part of this version. Preserve masked lazy lowering, bounded dense-list grouping, shared value caches, validated chunk offsets and the fieldless nullable-struct fix. Retain the diagnostic examples and regression benchmark workflow. Backtracking: previous PR head 8442d46; experimental CI head 60fda25; local pre-review checkpoint a253e5727 and reviewed checkpoint e8fd21a4a. The byte budget covers in-flight decoded work, not retained output or encoded/coalesced buffers. Full performance revalidation remains necessary. Validation: 692 layout and 79 scan library tests passed, followed by all 7 await protocol tests including an additional await/IO regression. Scoped all-target/all-feature clippy passed with -D warnings; formatted with nightly-2026-09-10. Focused E2E measurements follow this checkpoint. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Use the planner-driven V2 ScanBuilder directly for every Vortex random access measurement, including object-store reads. Remove the reader toggle, its configuration plumbing, V1 documentation, and unused reader-cache setup. Remove the experiment-only regression workflow, driver script, and three fixture/selection examples, along with their futures dev dependency. Keep the planner implementation and regression tests. The previous commit c4fa781 preserves the diagnostic harnesses and benchmark context. Inspected the resulting diff and searched for stale switch/example references. No tests, builds, or benchmarks were rerun for this cleanup. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

github-actions · 2026-10-19T10:36:24Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`fe25a60`	random-access-bench	1	Explore Profiling Data

Previous Runs (5)

Status	Commit	Job	Attempt	Link
🟢 Done	`1107f53`	random-access-bench	1	Explore Profiling Data
🟢 Done	`e3cbd8b`	random-access-bench	1	Explore Profiling Data
🟢 Done	`d72860e`	random-access-bench	2	Explore Profiling Data
🟢 Done	`a8349ec`	compress-bench	1	Explore Profiling Data
🟢 Done	`a8349ec`	random-access-bench	1	Explore Profiling Data

Powered by Polar Signals Cloud

github-actions · 2026-10-19T11:03:40Z

Benchmarks: Random Access 📖

Commits: PR fe25a609 vs base 92a7db07
Verdict: No clear signal (environment too noisy confidence)
Attributed Vortex impact: -8.5%
Engines: random-access No clear signal (-8.5%, environment too noisy confidence)
Vortex (geomean): hot 0.905x ➖
Parquet (geomean): hot 0.989x ➖
Shifts: Parquet (control) -1.1% · Median polish -5.2%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
Hot vs cold: Every measurement is run several times. The first run is reported as the cold run, and the median of the runs after it is reported as the hot run. The verdict and significance use hot runs; each target's geomean reports hot and cold beside each other where the individual runs were recorded and hot alone where they were not, the cold column shows first-run cost per row, and hot/cold is how much of each run the warm path saves. Rows whose results predate per-run reporting show only one value, taken from the value the runner reported.
Table cells: Each cell reads PR / base / %diff. The hot and cold columns mark a change as 🔴 slower or 🟢 faster once it clears this suite's threshold; anything smaller is noise here and is left unmarked, as is hot/cold, because a shift in the warm-up ratio is not a win or a loss by itself.

vortex / arrow-ipc / ns (1.066x ➖, 0↑ 2↓)

name	ns (PR / base / %diff)
random-access/arrow-tokio-local-disk	74300 / 68607 / +8.3%
random-access/arrow-tokio-local-disk-footer	171202 / 168527 / +1.6%
random-access/feature-vectors/correlated/arrow-tokio-local-disk	2472914 / 2522808 / -2.0%
random-access/feature-vectors/correlated/arrow-tokio-local-disk-footer	3298402 / 3493249 / -5.6%
random-access/feature-vectors/uniform/arrow-tokio-local-disk	31790857 / 30138993 / +5.5%
random-access/feature-vectors/uniform/arrow-tokio-local-disk-footer	55971021 / 32308217 / +73.2% 🔴
random-access/nested-lists/correlated/arrow-tokio-local-disk	63195 / 60045 / +5.2%
random-access/nested-lists/correlated/arrow-tokio-local-disk-footer	109365 / 104170 / +5.0%
random-access/nested-lists/uniform/arrow-tokio-local-disk	1138025 / 1078953 / +5.5%
random-access/nested-lists/uniform/arrow-tokio-local-disk-footer	1242659 / 1097061 / +13.3% 🔴
random-access/nested-structs/correlated/arrow-tokio-local-disk	57778 / 54612 / +5.8%
random-access/nested-structs/correlated/arrow-tokio-local-disk-footer	100158 / 94861 / +5.6%
random-access/nested-structs/uniform/arrow-tokio-local-disk	925858 / 903216 / +2.5%
random-access/nested-structs/uniform/arrow-tokio-local-disk-footer	969948 / 927192 / +4.6%
random-access/taxi/correlated/arrow-tokio-local-disk	127984 / 124134 / +3.1%
random-access/taxi/correlated/arrow-tokio-local-disk-footer	227980 / 225219 / +1.2%
random-access/taxi/uniform/arrow-tokio-local-disk	2877774 / 2813735 / +2.3%
random-access/taxi/uniform/arrow-tokio-local-disk-footer	2991190 / 2962933 / +1.0%

random-access / vortex-file-compressed / ns (0.905x ➖, 8↑ 1↓)

name	ns (PR / base / %diff)
random-access/feature-vectors/correlated/vortex-tokio-local-disk	180008 / 233649 / -23.0% 🟢
random-access/feature-vectors/correlated/vortex-tokio-local-disk-footer	1053993 / 1147837 / -8.2%
random-access/feature-vectors/uniform/vortex-tokio-local-disk	969808 / 1378299 / -29.6% 🟢
random-access/feature-vectors/uniform/vortex-tokio-local-disk-footer	2099397 / 2565372 / -18.2% 🟢
random-access/nested-lists/correlated/vortex-tokio-local-disk	160302 / 218309 / -26.6% 🟢
random-access/nested-lists/correlated/vortex-tokio-local-disk-footer	288647 / 353664 / -18.4% 🟢
random-access/nested-lists/uniform/vortex-tokio-local-disk	674040 / 677257 / -0.5%
random-access/nested-lists/uniform/vortex-tokio-local-disk-footer	894668 / 873786 / +2.4%
random-access/nested-structs/correlated/vortex-tokio-local-disk	222678 / 242718 / -8.3%
random-access/nested-structs/correlated/vortex-tokio-local-disk-footer	429764 / 437522 / -1.8%
random-access/nested-structs/uniform/vortex-tokio-local-disk	525047 / 586191 / -10.4% 🟢
random-access/nested-structs/uniform/vortex-tokio-local-disk-footer	806382 / 796521 / +1.2%
random-access/taxi/correlated/vortex-tokio-local-disk	638272 / 594014 / +7.5%
random-access/taxi/correlated/vortex-tokio-local-disk-footer	1204406 / 1199816 / +0.4%
random-access/taxi/uniform/vortex-tokio-local-disk	1757192 / 2343191 / -25.0% 🟢
random-access/taxi/uniform/vortex-tokio-local-disk-footer	2613876 / 3363875 / -22.3% 🟢
random-access/vortex-tokio-local-disk	511842 / 387324 / +32.1% 🔴
random-access/vortex-tokio-local-disk-footer	859368 / 871456 / -1.4%

random-access / parquet / ns (0.989x ➖, 1↑ 0↓)

name	ns (PR / base / %diff)
random-access/feature-vectors/correlated/parquet-tokio-local-disk	106375575976 / 106355839970 / +0.0%
random-access/feature-vectors/correlated/parquet-tokio-local-disk-footer	106872795882 / 113042205755 / -5.5%
random-access/feature-vectors/uniform/parquet-tokio-local-disk	106333250536 / 132160209457 / -19.5% 🟢
random-access/feature-vectors/uniform/parquet-tokio-local-disk-footer	107270567353 / 107764144489 / -0.5%
random-access/nested-lists/correlated/parquet-tokio-local-disk	148258232 / 143249847 / +3.5%
random-access/nested-lists/correlated/parquet-tokio-local-disk-footer	144368801 / 145152372 / -0.5%
random-access/nested-lists/uniform/parquet-tokio-local-disk	148039398 / 145444723 / +1.8%
random-access/nested-lists/uniform/parquet-tokio-local-disk-footer	144388986 / 147220794 / -1.9%
random-access/nested-structs/correlated/parquet-tokio-local-disk	44390748 / 44732303 / -0.8%
random-access/nested-structs/correlated/parquet-tokio-local-disk-footer	44347860 / 44823933 / -1.1%
random-access/nested-structs/uniform/parquet-tokio-local-disk	44356985 / 44715601 / -0.8%
random-access/nested-structs/uniform/parquet-tokio-local-disk-footer	44689474 / 45113025 / -0.9%
random-access/parquet-tokio-local-disk	121145109 / 119938138 / +1.0%
random-access/parquet-tokio-local-disk-footer	121422248 / 118538394 / +2.4%
random-access/taxi/correlated/parquet-tokio-local-disk	181691082 / 179834753 / +1.0%
random-access/taxi/correlated/parquet-tokio-local-disk-footer	181743651 / 177573135 / +2.3%
random-access/taxi/uniform/parquet-tokio-local-disk	190220503 / 192945410 / -1.4%
random-access/taxi/uniform/parquet-tokio-local-disk-footer	193162246 / 187342599 / +3.1%

random-access / lance / ns (1.004x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
random-access/feature-vectors/correlated/lance-tokio-local-disk	212011 / 209684 / +1.1%
random-access/feature-vectors/correlated/lance-tokio-local-disk-footer	980782 / 1009277 / -2.8%
random-access/feature-vectors/uniform/lance-tokio-local-disk	1069525 / 1065477 / +0.4%
random-access/feature-vectors/uniform/lance-tokio-local-disk-footer	1832227 / 1894783 / -3.3%
random-access/lance-tokio-local-disk	549823 / 548108 / +0.3%
random-access/lance-tokio-local-disk-footer	1318192 / 1317622 / +0.0%
random-access/nested-lists/correlated/lance-tokio-local-disk	165349 / 159403 / +3.7%
random-access/nested-lists/correlated/lance-tokio-local-disk-footer	496772 / 495934 / +0.2%
random-access/nested-lists/uniform/lance-tokio-local-disk	911960 / 915993 / -0.4%
random-access/nested-lists/uniform/lance-tokio-local-disk-footer	1409395 / 1423728 / -1.0%
random-access/nested-structs/correlated/lance-tokio-local-disk	286024 / 278613 / +2.7%
random-access/nested-structs/correlated/lance-tokio-local-disk-footer	563770 / 561231 / +0.5%
random-access/nested-structs/uniform/lance-tokio-local-disk	2335485 / 2270739 / +2.9%
random-access/nested-structs/uniform/lance-tokio-local-disk-footer	2595088 / 2646167 / -1.9%
random-access/taxi/correlated/lance-tokio-local-disk	796858 / 790786 / +0.8%
random-access/taxi/correlated/lance-tokio-local-disk-footer	1832241 / 1825701 / +0.4%
random-access/taxi/uniform/lance-tokio-local-disk	8222566 / 8031996 / +2.4%
random-access/taxi/uniform/lance-tokio-local-disk-footer	8858127 / 8661295 / +2.3%

github-actions · 2026-10-19T13:15:29Z

Benchmarks: Compression 📖

Commits: PR a8349ec4 vs base 92a7db07

vortex / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
compress time/Arade	790948193 / 798906660 / -1.0%
compress time/Bimbo	3495692253 / 3538601527 / -1.2%
compress time/CMSprovider	2501269245 / 2492648416 / +0.3%
compress time/Euro2016	264129948 / 262234961 / +0.7%
compress time/Food	313563631 / 315140162 / -0.5%
compress time/HashTags	509499170 / 513905990 / -0.9%
compress time/TPC-H l_comment canonical	601966973 / 604543222 / -0.4%
compress time/TPC-H l_comment chunked	601236887 / 602901965 / -0.3%
compress time/taxi	385798588 / 386340184 / -0.1%
compress time/wide table cols=100 chunks=1 rows=1000	4146733 / 4157958 / -0.3%
compress time/wide table cols=100 chunks=50 rows=1000	4191168 / 4168355 / +0.5%
compress time/wide table cols=1000 chunks=1 rows=1000	40944561 / 40984927 / -0.1%
compress time/wide table cols=1000 chunks=50 rows=1000	40563040 / 41563402 / -2.4%
compress time/wide table cols=10000 chunks=1 rows=1000	468795592 / 473560447 / -1.0%
compress time/wide table cols=10000 chunks=50 rows=1000	473042273 / 476546211 / -0.7%
decompress time/Arade	12044304 / 12279205 / -1.9%
decompress time/Bimbo	56419913 / 56820339 / -0.7%
decompress time/CMSprovider	37029368 / 36826112 / +0.6%
decompress time/Euro2016	11676279 / 11382773 / +2.6%
decompress time/Food	3279856 / 3308748 / -0.9%
decompress time/HashTags	35566258 / 35982177 / -1.2%
decompress time/TPC-H l_comment canonical	13308413 / 13160149 / +1.1%
decompress time/TPC-H l_comment chunked	13377104 / 13382017 / -0.0%
decompress time/taxi	8981322 / 8598412 / +4.5%
decompress time/wide table cols=100 chunks=1 rows=1000	1182900 / 1213153 / -2.5%
decompress time/wide table cols=100 chunks=50 rows=1000	1193261 / 1211679 / -1.5%
decompress time/wide table cols=1000 chunks=1 rows=1000	11872851 / 11786382 / +0.7%
decompress time/wide table cols=1000 chunks=50 rows=1000	11859218 / 11752035 / +0.9%
decompress time/wide table cols=10000 chunks=1 rows=1000	129100261 / 127408749 / +1.3%
decompress time/wide table cols=10000 chunks=50 rows=1000	128941670 / 127903083 / +0.8%

vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
vortex-file-compressed size/Arade	142213756 / 142213756 / +0.0%
vortex-file-compressed size/Bimbo	458581212 / 458581212 / +0.0%
vortex-file-compressed size/CMSprovider	420794836 / 420794836 / +0.0%
vortex-file-compressed size/Euro2016	164128948 / 164125572 / +0.0%
vortex-file-compressed size/Food	38950752 / 38950752 / +0.0%
vortex-file-compressed size/HashTags	196170516 / 196170516 / +0.0%
vortex-file-compressed size/TPC-H l_comment canonical	178688232 / 178688232 / +0.0%
vortex-file-compressed size/TPC-H l_comment chunked	178688232 / 178688232 / +0.0%
vortex-file-compressed size/taxi	49998092 / 49998092 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000	93277680 / 93277680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000	93277680 / 93277680 / +0.0%

vortex / vortex-file-compressed / ratio (1.000x ➖, 0↑ 0↓)

name	ratio (PR / base / %diff)
vortex:parquet-zstd ratio compress time/Arade	3.97145525 / 4.04805189 / -1.9%
vortex:parquet-zstd ratio compress time/Bimbo	3.58737853 / 3.62988296 / -1.2%
vortex:parquet-zstd ratio compress time/CMSprovider	5.12635912 / 5.18170599 / -1.1%
vortex:parquet-zstd ratio compress time/Euro2016	0.840162107 / 0.832523356 / +0.9%
vortex:parquet-zstd ratio compress time/Food	4.1347509 / 4.34381012 / -4.8%
vortex:parquet-zstd ratio compress time/HashTags	3.2709142 / 3.28491235 / -0.4%
vortex:parquet-zstd ratio compress time/TPC-H l_comment canonical	2.89196016 / 2.8853135 / +0.2%
vortex:parquet-zstd ratio compress time/TPC-H l_comment chunked	2.93989606 / 2.96612815 / -0.9%
vortex:parquet-zstd ratio compress time/taxi	4.17014987 / 4.13776056 / +0.8%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=1 rows=1000	3.26568409 / 3.17638754 / +2.8%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=50 rows=1000	3.20811496 / 3.27444984 / -2.0%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=1 rows=1000	2.94277621 / 3.05397198 / -3.6%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=50 rows=1000	3.14239742 / 3.10783476 / +1.1%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=1 rows=1000	3.34233476 / 3.41770928 / -2.2%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=50 rows=1000	3.44779899 / 3.44273564 / +0.1%
vortex:parquet-zstd ratio decompress time/Arade	0.243818436 / 0.248212341 / -1.8%
vortex:parquet-zstd ratio decompress time/Bimbo	0.623358398 / 0.624349443 / -0.2%
vortex:parquet-zstd ratio decompress time/CMSprovider	0.358888446 / 0.356339945 / +0.7%
vortex:parquet-zstd ratio decompress time/Euro2016	0.0779370346 / 0.0757707294 / +2.9%
vortex:parquet-zstd ratio decompress time/Food	0.130817887 / 0.131612312 / -0.6%
vortex:parquet-zstd ratio decompress time/HashTags	0.0848831275 / 0.0859493727 / -1.2%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment canonical	0.161000834 / 0.159037143 / +1.2%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment chunked	0.162934582 / 0.16198218 / +0.6%
vortex:parquet-zstd ratio decompress time/taxi	0.151560152 / 0.145478088 / +4.2%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=1 rows=1000	0.577600864 / 0.590342092 / -2.2%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=50 rows=1000	0.590047925 / 0.586533174 / +0.6%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=1 rows=1000	0.518212837 / 0.500118067 / +3.6%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=50 rows=1000	0.504868599 / 0.493777091 / +2.2%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=1 rows=1000	0.505653659 / 0.49776048 / +1.6%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=50 rows=1000	0.507896029 / 0.499295538 / +1.7%
vortex:parquet-zstd size/Arade	0.551185597 / 0.551185597 / -0.0%
vortex:parquet-zstd size/Bimbo	1.19261532 / 1.19261532 / +0.0%
vortex:parquet-zstd size/CMSprovider	1.11649863 / 1.11649863 / -0.0%
vortex:parquet-zstd size/Euro2016	1.33460902 / 1.33458157 / +0.0%
vortex:parquet-zstd size/Food	1.09107276 / 1.09107276 / -0.0%
vortex:parquet-zstd size/HashTags	1.46946859 / 1.46946859 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment canonical	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment chunked	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/taxi	0.904392267 / 0.904392267 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=1 rows=1000	1.00040408 / 1.00040408 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=50 rows=1000	1.00040408 / 1.00040408 / -0.0%

vortex / parquet / ns (0.999x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
parquet_rs-zstd compress time/Arade	199158279 / 197355835 / +0.9%
parquet_rs-zstd compress time/Bimbo	974441983 / 974852790 / -0.0%
parquet_rs-zstd compress time/CMSprovider	487923141 / 481047829 / +1.4%
parquet_rs-zstd compress time/Euro2016	314379744 / 314988113 / -0.2%
parquet_rs-zstd compress time/Food	75836160 / 72549249 / +4.5%
parquet_rs-zstd compress time/HashTags	155766596 / 156444354 / -0.4%
parquet_rs-zstd compress time/TPC-H l_comment canonical	208151890 / 209524276 / -0.7%
parquet_rs-zstd compress time/TPC-H l_comment chunked	204509573 / 203262278 / +0.6%
parquet_rs-zstd compress time/taxi	92514322 / 93369391 / -0.9%
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000	1269790 / 1309021 / -3.0%
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000	1306427 / 1272994 / +2.6%
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000	13913583 / 13420204 / +3.7%
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000	12908310 / 13373749 / -3.5%
parquet_rs-zstd compress time/wide table cols=10000 chunks=1 rows=1000	140259916 / 138560775 / +1.2%
parquet_rs-zstd compress time/wide table cols=10000 chunks=50 rows=1000	137201233 / 138420797 / -0.9%
parquet_rs-zstd decompress time/Arade	49398660 / 49470566 / -0.1%
parquet_rs-zstd decompress time/Bimbo	90509590 / 91007271 / -0.5%
parquet_rs-zstd decompress time/CMSprovider	103177933 / 103345450 / -0.2%
parquet_rs-zstd decompress time/Euro2016	149816824 / 150226520 / -0.3%
parquet_rs-zstd decompress time/Food	25071923 / 25140110 / -0.3%
parquet_rs-zstd decompress time/HashTags	419002681 / 418643858 / +0.1%
parquet_rs-zstd decompress time/TPC-H l_comment canonical	82660522 / 82748902 / -0.1%
parquet_rs-zstd decompress time/TPC-H l_comment chunked	82101073 / 82614131 / -0.6%
parquet_rs-zstd decompress time/taxi	59259125 / 59104516 / +0.3%
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000	2047954 / 2055000 / -0.3%
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000	2022312 / 2065832 / -2.1%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000	22911148 / 23567199 / -2.8%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000	23489712 / 23800284 / -1.3%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=1 rows=1000	255313610 / 255963971 / -0.3%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=50 rows=1000	253874145 / 256167086 / -0.9%

vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
parquet size/Arade	258014282 / 258014282 / +0.0%
parquet size/Bimbo	384517292 / 384517292 / +0.0%
parquet size/CMSprovider	376887911 / 376887911 / +0.0%
parquet size/Euro2016	122979049 / 122979049 / +0.0%
parquet size/Food	35699500 / 35699500 / +0.0%
parquet size/HashTags	133497590 / 133497590 / +0.0%
parquet size/TPC-H l_comment canonical	158358238 / 158358238 / +0.0%
parquet size/TPC-H l_comment chunked	158358238 / 158358238 / +0.0%
parquet size/taxi	55283635 / 55283635 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000	93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000	93240004 / 93240004 / +0.0%

vortex / arrow-ipc / ns (0.970x ➖, 5↑ 1↓)

name	ns (PR / base / %diff)
arrow-ipc compress time/Arade	99459789 / 100282127 / -0.8%
arrow-ipc compress time/Bimbo	615655924 / 620913325 / -0.8%
arrow-ipc compress time/CMSprovider	382300990 / 392187462 / -2.5%
arrow-ipc compress time/Euro2016	38153729 / 38513480 / -0.9%
arrow-ipc compress time/Food	33757053 / 34306794 / -1.6%
arrow-ipc compress time/HashTags	79145620 / 79232270 / -0.1%
arrow-ipc compress time/TPC-H l_comment canonical	938395784 / 936947403 / +0.2%
arrow-ipc compress time/TPC-H l_comment chunked	1165174199 / 937079164 / +24.3% 🔴
arrow-ipc compress time/taxi	74788588 / 75474858 / -0.9%
arrow-ipc compress time/wide table cols=100 chunks=1 rows=1000	200370 / 202997 / -1.3%
arrow-ipc compress time/wide table cols=100 chunks=50 rows=1000	198838 / 206833 / -3.9%
arrow-ipc compress time/wide table cols=1000 chunks=1 rows=1000	2224933 / 2264867 / -1.8%
arrow-ipc compress time/wide table cols=1000 chunks=50 rows=1000	2224404 / 2219266 / +0.2%
arrow-ipc compress time/wide table cols=10000 chunks=1 rows=1000	25191787 / 25553990 / -1.4%
arrow-ipc compress time/wide table cols=10000 chunks=50 rows=1000	25392235 / 25465709 / -0.3%
arrow-ipc decompress time/Arade	140766555 / 181138413 / -22.3% 🟢
arrow-ipc decompress time/Bimbo	416324043 / 428062968 / -2.7%
arrow-ipc decompress time/CMSprovider	472372121 / 581320789 / -18.7% 🟢
arrow-ipc decompress time/Euro2016	64042503 / 74596154 / -14.1% 🟢
arrow-ipc decompress time/Food	48836255 / 58361210 / -16.3% 🟢
arrow-ipc decompress time/HashTags	120553256 / 142887604 / -15.6% 🟢
arrow-ipc decompress time/TPC-H l_comment canonical	626713989 / 627522206 / -0.1%
arrow-ipc decompress time/TPC-H l_comment chunked	640669777 / 627672817 / +2.1%
arrow-ipc decompress time/taxi	45100991 / 48514220 / -7.0%
arrow-ipc decompress time/wide table cols=100 chunks=1 rows=1000	319051 / 319533 / -0.2%
arrow-ipc decompress time/wide table cols=100 chunks=50 rows=1000	318190 / 319011 / -0.3%
arrow-ipc decompress time/wide table cols=1000 chunks=1 rows=1000	3471763 / 3502025 / -0.9%
arrow-ipc decompress time/wide table cols=1000 chunks=50 rows=1000	3666629 / 3489886 / +5.1%
arrow-ipc decompress time/wide table cols=10000 chunks=1 rows=1000	37035555 / 36677966 / +1.0%
arrow-ipc decompress time/wide table cols=10000 chunks=50 rows=1000	36913908 / 36420684 / +1.4%

vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
arrow-ipc size/Arade	691765162 / 691765162 / +0.0%
arrow-ipc size/Bimbo	3279590114 / 3279590114 / +0.0%
arrow-ipc size/CMSprovider	2422259682 / 2422259682 / +0.0%
arrow-ipc size/Euro2016	360582666 / 360582666 / +0.0%
arrow-ipc size/Food	255782338 / 255782338 / +0.0%
arrow-ipc size/HashTags	704785810 / 704785810 / +0.0%
arrow-ipc size/TPC-H l_comment canonical	7317023586 / 7317023586 / +0.0%
arrow-ipc size/TPC-H l_comment chunked	7317023586 / 7317023586 / +0.0%
arrow-ipc size/taxi	482329690 / 482329690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000	125752506 / 125752506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000	125752506 / 125752506 / +0.0%

Add an opt-in prototype that compiles list element pipelines before offsets and validity resolve, then wakes their sources with the selected ranges. Keep the existing list implementation as the default and comparison control. Restore selection diagnostics and generate primitive, nested and short ListLayout fixtures. The random-access benchmark label now runs reference checks and alternating comparisons, with raw results uploaded as an artifact. Local validation: 72 reference checks and 32 timing cases, plus a reduced-iteration run of the CI comparison harness. This prototype does not yet implement the decoded-work admission budget or shared segment optimization. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

joseph-isaacs changed the title ~~perf(scan): compose selection planners and optimize V2 random access~~ Add selection planners and ListLayout scans to V2 Oct 16, 2026

joseph-isaacs force-pushed the ji/decompression-v2-scan branch from 6b97c79 to 15f2fed Compare October 16, 2026 16:52

joseph-isaacs changed the base branch from claude/planner-decode-slice-migration-xppbf1 to ji/upbeat-volta-adp0xd October 16, 2026 16:52

Base automatically changed from ji/upbeat-volta-adp0xd to develop October 16, 2026 16:57

joseph-isaacs force-pushed the ji/decompression-v2-scan branch from 15f2fed to 02119f6 Compare October 16, 2026 17:00

joseph-isaacs added the changelog/feature A new feature label Oct 16, 2026 — with ChatGPT Codex Connector

joseph-isaacs force-pushed the ji/decompression-v2-scan branch from 8442d46 to c4fa781 Compare October 19, 2026 08:43

joseph-isaacs changed the title ~~Add selection planners and ListLayout scans to V2~~ Add planner-owned selection and bounded ListLayout scans to V2 Oct 19, 2026

joseph-isaacs added the action/bench-random-access Run only the random-access benchmark on this PR label Oct 19, 2026