Skip to content

feat(array): add Narrow integer encoding - #10223

Draft
connortsui20 wants to merge 9 commits into
developfrom
ct/narrow-encoding
Draft

connortsui20 wants to merge 9 commits into
developfrom
ct/narrow-encoding

Conversation

@connortsui20

@connortsui20 connortsui20 commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Summary

Progress towards #10020. Related prototype: #10015.

Integer arrays can retain their logical dtype while storing a narrower child, so selecting rows and comparing values can use the smaller representation. This is also the first step towards giving canonical decimals an integer child whose logical width follows the decimal precision.

Changes

NarrowArray preserves signedness and nullability, accepts encoded children, and widens through a lazy cast when canonical primitive values are needed. Selection, compatible casts, fitting fill values, comparisons, and integer aggregates operate through the child. Aggregate partial states are converted back to the logical dtype, and arithmetic keeps the logical overflow behavior through canonical fallback.

This draft leaves compressor selection and decimal integration for follow-up work.

#10313 discusses whether dtype and range shortcuts may skip value-dependent errors from lazy children.

@codspeed

codspeed Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 26.7%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 29 improved benchmarks
✅ 2072 untouched benchmarks
🆕 3 new benchmarks
⏩ 518 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Simulation fsst_decompress_string 526 µs 188.4 µs ×2.8
⚡ Simulation chunked_canonicalize_into[(20, 100, 16, 4)] 739.9 µs 340.4 µs ×2.2
⚡ Simulation chunked_dict_fsst_into_canonical[(250, 10, 2)] 681.9 µs 342.6 µs +99.03%
⚡ Simulation decode_primitives[i64, (1000, 32)] 39.9 µs 26.1 µs +53.1%
⚡ Simulation chunked_canonicalize_into[(20, 50, 8, 4)] 274.4 µs 213 µs +28.85%
⚡ Simulation chunked_canonicalize_into[(20, 50, 8, 16)] 279.5 µs 217.7 µs +28.4%
⚡ Simulation chunked_canonicalize_into[(20, 50, 8, 64)] 306.3 µs 242.6 µs +26.29%
⚡ Simulation chunked_into_canonical[(20, 50, 8, 4)] 314.3 µs 252.2 µs +24.58%
⚡ Simulation chunked_into_canonical[(20, 50, 8, 16)] 323.3 µs 261 µs +23.87%
⚡ Simulation chunked_into_canonical[(20, 50, 8, 64)] 348.2 µs 285.6 µs +21.91%
⚡ Simulation decompress_fsst[(200, 4, 4)] 60.8 µs 50.6 µs +20.03%
⚡ Simulation decompress_fsst[(200, 4, 8)] 61.6 µs 51.6 µs +19.45%
⚡ Simulation chunked_canonicalize_into[(10, 150, 4, 4)] 235.3 µs 198.8 µs +18.34%
⚡ Simulation decompress_fsst[(200, 16, 4)] 66.4 µs 56.7 µs +17.14%
⚡ Simulation decompress_fsst[(200, 16, 8)] 69.4 µs 59.3 µs +16.97%
⚡ Simulation chunked_canonicalize_into[(10, 150, 16, 4)] 266.6 µs 230.1 µs +15.88%
⚡ Simulation chunked_canonicalize_into[(20, 100, 16, 16)] 461.3 µs 399.1 µs +15.59%
⚡ Simulation chunked_into_canonical[(10, 150, 4, 4)] 264.9 µs 230.3 µs +15.03%
⚡ Simulation chunked_into_canonical[(10, 150, 16, 4)] 298.7 µs 262.4 µs +13.82%
⚡ Simulation chunked_into_canonical[(20, 100, 16, 4)] 452.7 µs 397.9 µs +13.78%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/narrow-encoding (44780e8) with develop (3c87da0)2

Open in CodSpeed

Footnotes

  1. 518 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on develop (cc018ee) during the generation of this report, so 3c87da0 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@connortsui20
connortsui20 added this pull request to stack #10231 October 2, 2026 18:47
@connortsui20 connortsui20 added the changelog/feature A new feature label Oct 2, 2026
@connortsui20
connortsui20 removed this pull request from stack #10231 October 5, 2026 10:33
@connortsui20
connortsui20 changed the base branch from develop to ct/checked-lane-chunks October 5, 2026 10:34
@connortsui20
connortsui20 added this pull request to stack #10300 October 5, 2026 10:34
@connortsui20
connortsui20 force-pushed the ct/narrow-encoding branch 3 times, most recently from 1d87c9c to 2be6820 Compare October 5, 2026 11:12
@connortsui20
connortsui20 removed this pull request from stack #10300 October 5, 2026 15:21
@connortsui20
connortsui20 changed the base branch from ct/checked-lane-chunks to develop October 5, 2026 15:26
@connortsui20
connortsui20 added this pull request to stack #10307 October 5, 2026 15:26
@connortsui20
connortsui20 removed this pull request from stack #10307 October 6, 2026 13:09
@connortsui20
connortsui20 added this pull request to stack #10334 October 6, 2026 13:10
@connortsui20
connortsui20 removed this pull request from stack #10334 October 6, 2026 13:44
@connortsui20
connortsui20 added this pull request to stack #10335 October 6, 2026 13:44
@connortsui20
connortsui20 removed this pull request from stack #10335 October 6, 2026 14:12
@connortsui20
connortsui20 added this pull request to stack #10336 October 6, 2026 14:17
@connortsui20
connortsui20 removed this pull request from stack #10336 October 6, 2026 14:18
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A valid nullable fill scalar still requires a nullable result dtype. Preserve that tag when casting the fill scalar into the stored child dtype.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 added this pull request to stack #10337 October 6, 2026 14:23

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant