Repository navigation
feat(compressor): measure inputs and blocks by the bytes their rows reference - #10445
Conversation
…ure blocks exactly
Arrow's Parquet reader points each batch's string views into the whole
decompressed page, and VarBinView::slice keeps every buffer, so the
nbytes() of an 8,192-row block of TPC-H l_comment was about 1 MiB for
27 KiB of strings. The repartition writer measured blocks with nbytes(),
so every block already exceeded the 1 MiB minimum and never coalesced.
Rather than trimming buffers on import, which costs a pass over every
imported view array, keep the import zero-copy and let callers that
size blocks ask for a precise measurement:
- `UncompressedSizeInBytes` takes `UncompressedSizeOpts { precision }`.
`Inexact` (the default) keeps today's behaviour and counts every
held buffer. `Exact` counts only the bytes the valid rows reference:
a VarBinView walks its views, a ListView rebuilds to its referenced
elements, nested types forward the precision to their children, and
fixed-width types are the same either way.
- The `Stat::UncompressedSizeInBytes` slot is the exact size. Only the
exact aggregate reads and writes it as `Precision::Exact`; an inexact
measurement is cached as `Precision::Inexact`, an upper bound that
inexact queries reuse and exact queries ignore. `compute_stat`, and so
file statistics, now measure exactly.
- The repartition coalescer measures each block exactly, so view blocks
sharing a large buffer coalesce up to `block_size_minimum`.
Adds a regression test streaming ten 8,192-row view blocks that share a
2 MiB buffer and asserting they coalesce three to a block.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…tes from uncompressed output The writer caches the exact uncompressed size on every chunk before compression. The compressor now reads that size, computing it without recording anything where it is absent, as the baseline a scheme has to beat, for the full array and for the sampled slices whose views still point at the full array's buffers. Dead bytes in a view array's buffers no longer make a dictionary of distinct strings look like a 20x win. Compaction moves from before scheme selection to after it. A scheme that changes the encoding rebuilds the data it keeps, so its dead bytes are gone for free. An array that keeps its canonical encoding, because no scheme matched, won, or beat the exact size, is compacted when it holds more than its rows reference: buffers used end to end are kept, contiguous ranges are sliced without copying, and the rest rewritten. The previous up-front compaction skipped arrays holding at most 128 bytes per row or with over half their bytes referenced, which let an Arrow page shared by 8,192-row blocks pass through whole. `uncompressed_size_in_bytes_uncached` measures columnar arrays without caching a statistic on them or their children, since an uncompressed leaf is serialized with its statistics and a size the compressor measured for itself would grow the file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…fragments The uncompressed size aggregate does not apply to variant arrays, so the coalescer, the compressing layout and the compressor fell over on the first variant chunk. Each now checks the dtype first and sizes an unsupported array by the bytes it holds, as before. The Python dataset test counted 26 fragments, which were the row splits left by string blocks that never coalesced. Measured exactly, the string column's blocks line up with the numeric columns' and the file has 16. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…ange The compressor baseline and the layout coalescing move to their own changes stacked on this one, and the longest comments are shortened. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…eference A scheme now competes against the exact uncompressed size of the array, not the buffers it holds, so a view array sharing a decompressed Parquet page no longer makes every scheme look like a win. The repartition writer coalesces blocks by the same measure, so such blocks still reach the block size minimum, and the compressing layout caches the exact size on each chunk for the compressor to read. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will improve performance by 39.37%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | threshold_neon[80] |
7.7 µs | 4.5 µs | +68.87% |
| ⚡ | WallTime | threshold_neon[64] |
6.9 µs | 4.2 µs | +65.81% |
| ⚡ | WallTime | threshold_neon[8] |
4.6 µs | 2.8 µs | +64.62% |
| ⚡ | WallTime | filter_neon[64] |
4.6 µs | 3 µs | +56.11% |
| ⚡ | WallTime | bitpack_blocked_compress_avx2 |
7.6 µs | 6.7 µs | +13.4% |
| ⚡ | WallTime | filter_neon[80] |
3.6 µs | 3.2 µs | +12.46% |
| ⚡ | WallTime | threshold_avx512[8] |
2.1 µs | 1.9 µs | +11.29% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/exact-size-compressor-baseline (7d66806) with develop (84e0d73)
…e flag The exact size of a list view's elements is found by scanning offsets and sizes: lists that tile one contiguous run are measured as a slice of it, fixed-width elements are sized by count, and only overlapping or gapped lists of variable-width elements are rebuilt. The accumulator already caches an exact result through the stat slot, so the cache flag and the uncached variant go. Precision is documented as approximate versus actual bytes used. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: 729502c I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: 2d03521 I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: b840fc0 I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: 09ef167 Signed-off-by: Claude <noreply@anthropic.com>
The uncached variant is gone: the writer caches the exact size on each chunk anyway, so the compressor reads and records through the one path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: 3402fe7 Signed-off-by: Claude <noreply@anthropic.com>
…en file The writer now records the exact uncompressed size on each chunk before compression, and the array statistics serializer writes it with the other statistics, so the stock ticker file grows by 24 bytes. The compact strategy writes no such statistics and is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…ressor-baseline Signed-off-by: Claude <noreply@anthropic.com>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
commented
Oct 12, 2026
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
Totals:
|
commented
Oct 12, 2026
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.974x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (0.972x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.009x ➖, 3↑ 2↓)
duckdb / vortex-compact / ns (1.011x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.990x ➖, 0↑ 0↓)
File Size Changes (2 files changed, +0.0% overall, 1↑ 1↓)
Totals:
|
commented
Oct 12, 2026
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.010x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.013x ➖, 0↑ 2↓)
duckdb / vortex-compact / ns (1.032x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
File Size Changes (14 files changed, -0.4% overall, 3↑ 11↓)
Totals:
|
commented
Oct 12, 2026
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.016x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 1↑ 1↓)
duckdb / vortex-compact / ns (0.984x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
File Size Changes (200 files changed, -0.0% overall, 99↑ 101↓)
Totals:
|
commented
Oct 12, 2026
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.082x ➖, 0↑ 3↓)
datafusion / parquet / ns (0.841x ➖, 4↑ 0↓)
duckdb / vortex-file-compressed / ns (1.081x ➖, 0↑ 2↓)
duckdb / vortex-compact / ns (0.914x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.938x ➖, 0↑ 0↓)
|
commented
Oct 12, 2026
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
File Size Changes (5 files changed, -0.0% overall, 2↑ 3↓)
Totals:
|
commented
Oct 12, 2026
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.014x ➖, 2↑ 2↓)
duckdb / vortex-compact / ns (1.006x ➖, 1↑ 3↓)
duckdb / parquet / ns (1.001x ➖, 1↑ 1↓)
File Size Changes (200 files changed, +0.0% overall, 122↑ 78↓)
Totals:
|
commented
Oct 12, 2026
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 1↑ 3↓)
duckdb / vortex-compact / ns (0.999x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.998x ➖, 5↑ 3↓)
File Size Changes (42 files changed, +0.0% overall, 30↑ 12↓)
Totals:
|
commented
Oct 12, 2026
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.007x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.973x ➖, 2↑ 0↓)
duckdb / vortex-compact / ns (0.995x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.988x ➖, 0↑ 0↓)
File Size Changes (14 files changed, -0.3% overall, 3↑ 11↓)
Totals:
|
commented
Oct 12, 2026
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.984x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (0.990x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
File Size Changes (2 files changed, +0.0% overall, 2↑ 0↓)
Totals:
|
commented
Oct 12, 2026
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ms (0.812x ✅, 6↑ 0↓)
vortex / vortex-file-compressed / % (0.952x ➖, 1↑ 0↓)
|
commented
Oct 12, 2026
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.950x ➖, 1↑ 1↓)
datafusion / vortex-compact / ns (0.747x ➖, 7↑ 0↓)
datafusion / parquet / ns (0.950x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.020x ➖, 0↑ 2↓)
duckdb / vortex-compact / ns (0.897x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.900x ➖, 1↑ 0↓)
|
commented
Oct 12, 2026
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.963x ➖, 3↑ 2↓)
vortex / vortex-file-compressed / bytes (1.002x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.976x ➖, 3↑ 2↓)
vortex / parquet / ns (1.000x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (1.018x ➖, 0↑ 4↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
commented
Oct 12, 2026
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.131x ➖, 0↑ 6↓)
datafusion / parquet / ns (1.026x ➖, 1↑ 3↓)
duckdb / vortex-compact / ns (1.059x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.184x ➖, 0↑ 5↓)
|
commented
Oct 12, 2026
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (0.999x ➖, 1↑ 2↓)
random-access / vortex-file-compressed / ns (0.985x ➖, 0↑ 0↓)
random-access / parquet / ns (1.000x ➖, 1↑ 1↓)
random-access / lance / ns (0.993x ➖, 0↑ 0↓)
|
commented
Oct 12, 2026
Benchmarks: Random Access (S3) 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (0.964x ➖, 3↑ 2↓)
random-access / parquet / ns (1.030x ➖, 0↑ 1↓)
random-access / lance / ns (1.166x ➖, 0↑ 4↓)
|
…ressor-baseline Signed-off-by: Claude <noreply@anthropic.com> # Conflicts: # vortex-file/src/tests.rs
Summary
Second of three stacked PRs, on top of vortex-data/vortex#10439. Uses the exact uncompressed size where
nbytes()overstated view arrays that share a decompressed Parquet page.Changes
CascadingCompressormeasures the input, and each leaf a scheme competes against, by the bytes its rows reference. A dictionary of distinct strings no longer looks like a 20x win against a 2 MiB shared page. Sampling scores slices the same way.block_size_minimuminstead of each being emitted alone. The Python dataset test's fragment count drops from 26 to 16 as the string column's blocks now line up with the numeric columns'.🤖 Generated with Claude Code
https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst
Generated by Claude Code
Generated by Claude Code