Repository navigation
feat(array): add precision option to uncompressed_size_in_bytes - #10439
Merged
Merged
Conversation
…ure blocks exactly
Arrow's Parquet reader points each batch's string views into the whole
decompressed page, and VarBinView::slice keeps every buffer, so the
nbytes() of an 8,192-row block of TPC-H l_comment was about 1 MiB for
27 KiB of strings. The repartition writer measured blocks with nbytes(),
so every block already exceeded the 1 MiB minimum and never coalesced.
Rather than trimming buffers on import, which costs a pass over every
imported view array, keep the import zero-copy and let callers that
size blocks ask for a precise measurement:
- `UncompressedSizeInBytes` takes `UncompressedSizeOpts { precision }`.
`Inexact` (the default) keeps today's behaviour and counts every
held buffer. `Exact` counts only the bytes the valid rows reference:
a VarBinView walks its views, a ListView rebuilds to its referenced
elements, nested types forward the precision to their children, and
fixed-width types are the same either way.
- The `Stat::UncompressedSizeInBytes` slot is the exact size. Only the
exact aggregate reads and writes it as `Precision::Exact`; an inexact
measurement is cached as `Precision::Inexact`, an upper bound that
inexact queries reuse and exact queries ignore. `compute_stat`, and so
file statistics, now measure exactly.
- The repartition coalescer measures each block exactly, so view blocks
sharing a large buffer coalesce up to `block_size_minimum`.
Adds a regression test streaming ten 8,192-row view blocks that share a
2 MiB buffer and asserting they coalesce three to a block.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will improve performance by 14.77%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | bitpack_blocked_compress_avx512 |
5.6 µs | 4.9 µs | +14.77% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/bold-gates-h5cp86 (a4d80dd) with develop (ea4ba3a)
joseph-isaacs
added
the
action/bench-compress
github-actions
Bot
removed
the
action/bench-compress
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (1)
Powered by Polar Signals Cloud |
…tes from uncompressed output The writer caches the exact uncompressed size on every chunk before compression. The compressor now reads that size, computing it without recording anything where it is absent, as the baseline a scheme has to beat, for the full array and for the sampled slices whose views still point at the full array's buffers. Dead bytes in a view array's buffers no longer make a dictionary of distinct strings look like a 20x win. Compaction moves from before scheme selection to after it. A scheme that changes the encoding rebuilds the data it keeps, so its dead bytes are gone for free. An array that keeps its canonical encoding, because no scheme matched, won, or beat the exact size, is compacted when it holds more than its rows reference: buffers used end to end are kept, contiguous ranges are sliced without copying, and the rest rewritten. The previous up-front compaction skipped arrays holding at most 128 bytes per row or with over half their bytes referenced, which let an Arrow page shared by 8,192-row blocks pass through whole. `uncompressed_size_in_bytes_uncached` measures columnar arrays without caching a statistic on them or their children, since an uncompressed leaf is serialized with its statistics and a size the compressor measured for itself would grow the file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Contributor
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.948x ➖, 3↑ 1↓)
vortex / vortex-file-compressed / bytes (0.999x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.963x ➖, 3↑ 1↓)
vortex / parquet / ns (1.003x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (1.010x ➖, 0↑ 2↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
…fragments The uncompressed size aggregate does not apply to variant arrays, so the coalescer, the compressing layout and the compressor fell over on the first variant chunk. Each now checks the dtype first and sizes an unsupported array by the bytes it holds, as before. The Python dataset test counted 26 fragments, which were the row splits left by string blocks that never coalesced. Measured exactly, the string column's blocks line up with the numeric columns' and the file has 16. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
joseph-isaacs
enabled auto-merge (squash)
October 12, 2026 11:01
robert3005
reviewed
Oct 12, 2026
| /// shared with other arrays, such as the decompressed pages Arrow's Parquet reader hands out, or | ||
| /// left behind by slicing and taking. Buffers used end to end are kept, contiguous used ranges | ||
| /// are sliced without copying, and the rest are rewritten. | ||
| fn without_dead_bytes( |
Contributor
There was a problem hiding this comment.
this seems to be weirdly placed - maybe this can move closer to the binary/utf8?
Contributor
Author
There was a problem hiding this comment.
Moved into the Canonical::VarBinView arm of the dispatch, right after choose_and_compress, so the string and binary handling owns it. That change now lives in the third PR of the stack, vortex-data/vortex#10446; this PR is down to the precision option on the aggregate.
Generated by Claude Code
robert3005
reviewed
Oct 12, 2026
|
|
||
| let canonical = child.clone().execute::<CanonicalValidity>(exec_ctx)?.0; | ||
| let compact = canonical.compact(exec_ctx)?; | ||
| let trimmed = trim(canonical, exec_ctx)?; |
Contributor
There was a problem hiding this comment.
maybe trim is an option to Canonical::compact?
Contributor
Author
There was a problem hiding this comment.
Done in the third PR of the stack, vortex-data/vortex#10446, commit 52eb242: Canonical::compact_with(CompactOptions) with CompactOptions::trim() skipping only the view buffer rewrite, and compact unchanged as the default. The compressor's separate trim is gone.
Generated by Claude Code
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.989x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.933x ➖, 3↑ 3↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.017x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
File Size Changes (6 files changed, -0.6% overall, 0↑ 6↓)
Totals:
|
…ange The compressor baseline and the layout coalescing move to their own changes stacked on this one, and the longest comments are shortened. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.991x ➖, 1↑ 1↓)
File Size Changes (99 files changed, -0.0% overall, 14↑ 85↓)
Totals:
|
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 1↑ 3↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 3↓)
File Size Changes (12 files changed, -0.0% overall, 5↑ 7↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.968x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 1↓)
File Size Changes (6 files changed, -0.5% overall, 1↑ 5↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.056x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.018x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (1.100x ➖, 0↑ 4↓)
duckdb / parquet / ns (1.039x ➖, 0↑ 0↓)
|
joseph-isaacs
changed the title
robert3005
approved these changes
Oct 12, 2026
…e flag The exact size of a list view's elements is found by scanning offsets and sizes: lists that tile one contiguous run are measured as a slice of it, fixed-width elements are sized by count, and only overlapping or gapped lists of variable-width elements are rebuilt. The accumulator already caches an exact result through the stat slot, so the cache flag and the uncached variant go. Precision is documented as approximate versus actual bytes used. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: 729502c I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: 2d03521 I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: b840fc0 I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: 09ef167 Signed-off-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Arrow's Parquet reader points each batch's
Utf8View/BinaryViewviews into the whole decompressed page, andVarBinView::slicekeeps every buffer.nbytes()sums full buffer lengths, so an 8,192-row block of TPC-Hl_commentreports about 1 MiB for 27 KiB of strings. The writer and compressor need a measure of the bytes the rows reference.First of three stacked PRs:
Changes
UncompressedSizeInBytestakesUncompressedSizeOpts { precision }.Inexactis the default and keeps the current behaviour of counting every buffer the canonical array holds.Exactcounts only the bytes the valid rows reference: aVarBinViewwalks its views, aListViewrebuilds to its referenced elements, nested types forward the precision to their children, and fixed-width types are identical either way.UncompressedSizeInBytesOptsprotobuf message carries the option through serialization.UncompressedSizeInBytesstat slot holds the exact size. An inexact result is cached as an upper bound that only inexact queries reuse.uncompressed_size_in_bytes_uncachedmeasures without recording a statistic, for arrays whose metadata is about to be serialized.🤖 Generated with Claude Code
https://claude.ai/code/session_015LWq6yREM1uAD9mbD2ZAst
Generated by Claude Code