Repository navigation
feat(fsst): Compute uncompressed size without decoding - #9755
lwwmanning wants to merge 1 commit into
Conversation
Signed-off-by: Will Manning <will@willmanning.io>
Merging this PR will improve performance by 14.57%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | decompress[u64, (4000, 1024)] |
85.7 µs | 70.4 µs | +21.81% |
| ⚡ | Simulation | allocate_drop_bytes[0] |
520.2 ns | 466 ns | +11.62% |
| ⚡ | WallTime | mul_u32_nonnull_avx512 |
6.3 µs | 5.7 µs | +10.61% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing wm/fsst-uncompressed-size (655c3dd) with develop (dab1684)
Footnotes
-
206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
| fn checked_sum_lengths<P>(lengths: &[P]) -> VortexResult<u64> | ||
| where | ||
| P: Copy, | ||
| u64: TryFrom<P>, | ||
| <u64 as TryFrom<P>>::Error: std::fmt::Display, | ||
| { | ||
| lengths.iter().try_fold(0u64, |total, length| { | ||
| let length = u64::try_from(*length).map_err(|error| { | ||
| vortex_err!("Failed to convert FSST uncompressed length to u64: {error}") | ||
| })?; | ||
| total | ||
| .checked_add(length) | ||
| .ok_or_else(|| vortex_err!("FSST decoded size overflowed u64")) | ||
| }) | ||
| } | ||
|
|
There was a problem hiding this comment.
does this need to be checked?
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
This branch has not been deployed
3 participants
Summary
FSST already stores each value’s uncompressed length. This adds an FSST-specific
UncompressedSizeInBytesaggregate that calculates the canonical size without decoding string payloads.Changes
The kernel reads only the lengths child and accounts for canonical views and validity. It rejects negative lengths and arithmetic overflow. Tests compare results with materialized UTF-8 and binary arrays. They cover nullable, all-null, and sliced inputs.
🤖 Generated with Codex