Skip to content

feat: add bitpacked v2 serialization - #9942

Closed
mhk197 wants to merge 1 commit into
mk/bitpacked-stack-05-explicit-packingfrom
mk/bitpacked-stack-06-v2-wire
Closed

mhk197 wants to merge 1 commit into
mk/bitpacked-stack-05-explicit-packingfrom
mk/bitpacked-stack-06-v2-wire

Conversation

@mhk197

@mhk197 mhk197 commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Serialize arrays with differing chunk widths under fastlanes.bitpacked_v2. The format stores one offsets child containing num_chunks + 1 byte boundaries; widths are derived from adjacent differences. Metadata contains only bounded array and patch information.

Uniform arrays retain the v1 wire format, which omits the offsets child and stores one width in metadata. Empty arrays use width zero because there are no chunks from which to derive a width. Deserialization still accepts legacy empty arrays with a nonzero width.

Add recursive round trips, execution with compressed offsets, validation of malformed compressed boundaries, and rejection of mismatched format IDs and child layouts. Extract shared wire-format helpers while preserving the v1 validation order. CUDA support remains limited to uniform widths.

Validation: 403 FastLanes/BtrBlocks tests passed (1 skipped).

@codspeed

codspeed Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 79.37%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 4 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚡ 1 improved benchmark
✅ 2231 untouched benchmarks
⏩ 329 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Simulation take_fsl_random[128, 10] 58.6 µs 32.7 µs +79.37%
⚠️ Simulation take_fsl_u32_random[256, 10] < 1 ns < 1 ns N/A
⚠️ Simulation fixed_16_advancing_ptr_safe[100] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[1000] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[10000] < 1 ns < 1 ns N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/bitpacked-stack-06-v2-wire (60ce538) with mk/bitpacked-stack-05-explicit-packing (0e28beb)2

Open in CodSpeed

Footnotes

  1. 329 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on mk/bitpacked-stack-05-explicit-packing (82c7619) during the generation of this report, so a3ce2bd was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@mhk197
mhk197 added this pull request to stack #9946 September 18, 2026 19:27
@mhk197
mhk197 force-pushed the mk/bitpacked-stack-06-v2-wire branch from 899c596 to db2c61b Compare September 18, 2026 19:42
@mhk197
mhk197 deployed to duckdb-build September 18, 2026 19:42 — with GitHub Actions Active
@mhk197
mhk197 deployed to duckdb-build September 18, 2026 19:42 — with GitHub Actions Active
@mhk197
mhk197 deployed to duckdb-build September 18, 2026 19:42 — with GitHub Actions Active
@mhk197
mhk197 force-pushed the mk/bitpacked-stack-06-v2-wire branch 2 times, most recently from 377f70c to e265aeb Compare September 23, 2026 17:31
@mhk197
mhk197 force-pushed the mk/bitpacked-stack-05-explicit-packing branch from b055b6b to a90e6fe Compare September 23, 2026 17:31
@mhk197
mhk197 force-pushed the mk/bitpacked-stack-06-v2-wire branch from e265aeb to 4b029d8 Compare September 23, 2026 18:17
@mhk197
mhk197 force-pushed the mk/bitpacked-stack-05-explicit-packing branch from a90e6fe to 0e28beb Compare September 23, 2026 18:17
@mhk197
mhk197 removed this pull request from stack #9946 September 23, 2026 18:19
@mhk197
mhk197 added this pull request to stack #10006 September 23, 2026 18:22
Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197
mhk197 removed this pull request from stack #10006 September 23, 2026 19:43
@mhk197
mhk197 added this pull request to stack #10012 September 23, 2026 19:43
@mhk197
mhk197 force-pushed the mk/bitpacked-stack-05-explicit-packing branch from 0e28beb to 82c7619 Compare September 23, 2026 19:43
@mhk197
mhk197 force-pushed the mk/bitpacked-stack-06-v2-wire branch from 4b029d8 to 60ce538 Compare September 23, 2026 19:43
@mhk197 mhk197 closed this Oct 2, 2026
@mhk197
mhk197 removed this pull request from stack #10012 October 2, 2026 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant