Skip to content

feat(compress): retain Narrow integer children through compression - #10333

Draft
connortsui20 wants to merge 4 commits into
ct/narrow-pushdownfrom
ct/narrow-compression
Draft

connortsui20 wants to merge 4 commits into
ct/narrow-pushdownfrom
ct/narrow-compression

Conversation

@connortsui20

@connortsui20 connortsui20 commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Summary

Progress towards #10020. Stacked on #10332.

Compressing a narrow integer child lets codecs decode at the smaller width while a Narrow wrapper retains the column's logical type. Session presets now use this representation when preview2026.10.0 is enabled. Core-only and CUDA presets keep it disabled.

Changes

The old primitive narrow method is replaced by shared width selection in NarrowArray. Internal buffers whose dtypes are chosen at construction use encode_values; arrays with a logical dtype contract use encode. Both retain signedness, so nonnegative signed internal buffers no longer silently become unsigned.

API Changes

Replaces PrimitiveArrayExt::narrow with NarrowArray::encode_values for internal buffers. Use NarrowArray::encode when the logical dtype must be retained.

@connortsui20 connortsui20 added the changelog/break A breaking API change label Oct 6, 2026
@connortsui20
connortsui20 added this pull request to stack #10334 October 6, 2026 13:10
@codspeed

codspeed Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 3 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 10 improved benchmarks
❌ 3 regressed benchmarks
✅ 2091 untouched benchmarks
⏩ 518 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation take_map[(0.1, 1.0)] 225.3 µs 289.3 µs -22.12%
❌ WallTime bitpack_blocked_compress_avx2 6.7 µs 7.6 µs -11.67%
❌ Simulation take_map[(0.1, 0.5)] 147.7 µs 166.8 µs -11.44%
⚡ Simulation encode_primitives[u8, (1000, 2)] 117.3 µs 74 µs +58.52%
⚡ Simulation encode_primitives[u8, (1000, 4)] 117.5 µs 74.2 µs +58.36%
⚡ Simulation encode_primitives[u8, (1000, 8)] 118.7 µs 75.3 µs +57.72%
⚡ Simulation encode_primitives[u8, (1000, 32)] 120.1 µs 77 µs +55.97%
⚡ Simulation encode_primitives[u8, (1000, 512)] 136.1 µs 92.8 µs +46.68%
⚡ Simulation encode_primitives[u8, (2000, 4)] 169.4 µs 125.9 µs +34.54%
⚡ Simulation encode_primitives[u8, (2000, 2)] 169.2 µs 125.8 µs +34.48%
⚡ Simulation encode_primitives[u8, (2000, 8)] 170.1 µs 126.9 µs +34.09%
⚡ Simulation encode_primitives[u8, (2000, 32)] 172.1 µs 128.8 µs +33.6%
⚡ Simulation encode_primitives[u8, (2000, 512)] 188.4 µs 145 µs +29.91%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/narrow-compression (71f9458) with ct/narrow-pushdown (557a613)

Open in CodSpeed

Footnotes

  1. 518 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@connortsui20
connortsui20 removed this pull request from stack #10334 October 6, 2026 13:44
@connortsui20
connortsui20 force-pushed the ct/narrow-compression branch from 76da12f to 756d7f0 Compare October 6, 2026 13:44
@connortsui20
connortsui20 added this pull request to stack #10335 October 6, 2026 13:44
@connortsui20
connortsui20 removed this pull request from stack #10335 October 6, 2026 14:12
@connortsui20
connortsui20 force-pushed the ct/narrow-compression branch from 756d7f0 to 0310216 Compare October 6, 2026 14:15
@connortsui20
connortsui20 added this pull request to stack #10336 October 6, 2026 14:17
@connortsui20
connortsui20 removed this pull request from stack #10336 October 6, 2026 14:18
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/narrow-compression branch from b641848 to 71f9458 Compare October 6, 2026 14:21
@connortsui20
connortsui20 added this pull request to stack #10337 October 6, 2026 14:23

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant