Repository navigation
feat(array): add Narrow integer encoding - #10223
connortsui20 wants to merge 9 commits into
Conversation
Merging this PR will improve performance by 26.7%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | fsst_decompress_string |
526 µs | 188.4 µs | ×2.8 |
| ⚡ | Simulation | chunked_canonicalize_into[(20, 100, 16, 4)] |
739.9 µs | 340.4 µs | ×2.2 |
| ⚡ | Simulation | chunked_dict_fsst_into_canonical[(250, 10, 2)] |
681.9 µs | 342.6 µs | +99.03% |
| ⚡ | Simulation | decode_primitives[i64, (1000, 32)] |
39.9 µs | 26.1 µs | +53.1% |
| ⚡ | Simulation | chunked_canonicalize_into[(20, 50, 8, 4)] |
274.4 µs | 213 µs | +28.85% |
| ⚡ | Simulation | chunked_canonicalize_into[(20, 50, 8, 16)] |
279.5 µs | 217.7 µs | +28.4% |
| ⚡ | Simulation | chunked_canonicalize_into[(20, 50, 8, 64)] |
306.3 µs | 242.6 µs | +26.29% |
| ⚡ | Simulation | chunked_into_canonical[(20, 50, 8, 4)] |
314.3 µs | 252.2 µs | +24.58% |
| ⚡ | Simulation | chunked_into_canonical[(20, 50, 8, 16)] |
323.3 µs | 261 µs | +23.87% |
| ⚡ | Simulation | chunked_into_canonical[(20, 50, 8, 64)] |
348.2 µs | 285.6 µs | +21.91% |
| ⚡ | Simulation | decompress_fsst[(200, 4, 4)] |
60.8 µs | 50.6 µs | +20.03% |
| ⚡ | Simulation | decompress_fsst[(200, 4, 8)] |
61.6 µs | 51.6 µs | +19.45% |
| ⚡ | Simulation | chunked_canonicalize_into[(10, 150, 4, 4)] |
235.3 µs | 198.8 µs | +18.34% |
| ⚡ | Simulation | decompress_fsst[(200, 16, 4)] |
66.4 µs | 56.7 µs | +17.14% |
| ⚡ | Simulation | decompress_fsst[(200, 16, 8)] |
69.4 µs | 59.3 µs | +16.97% |
| ⚡ | Simulation | chunked_canonicalize_into[(10, 150, 16, 4)] |
266.6 µs | 230.1 µs | +15.88% |
| ⚡ | Simulation | chunked_canonicalize_into[(20, 100, 16, 16)] |
461.3 µs | 399.1 µs | +15.59% |
| ⚡ | Simulation | chunked_into_canonical[(10, 150, 4, 4)] |
264.9 µs | 230.3 µs | +15.03% |
| ⚡ | Simulation | chunked_into_canonical[(10, 150, 16, 4)] |
298.7 µs | 262.4 µs | +13.82% |
| ⚡ | Simulation | chunked_into_canonical[(20, 100, 16, 4)] |
452.7 µs | 397.9 µs | +13.78% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/narrow-encoding (44780e8) with develop (3c87da0)2
Footnotes
-
518 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
develop(cc018ee) during the generation of this report, so 3c87da0 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
8d99b2f to
6b73e11
Compare
1d87c9c to
2be6820
Compare
2be6820 to
42df1b7
Compare
42df1b7 to
91d8573
Compare
91d8573 to
9ee6363
Compare
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A valid nullable fill scalar still requires a nullable result dtype. Preserve that tag when casting the fill scalar into the stored child dtype. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
9ee6363 to
44780e8
Compare
Summary
Progress towards #10020. Related prototype: #10015.
Integer arrays can retain their logical dtype while storing a narrower child, so selecting rows and comparing values can use the smaller representation. This is also the first step towards giving canonical decimals an integer child whose logical width follows the decimal precision.
Changes
NarrowArraypreserves signedness and nullability, accepts encoded children, and widens through a lazy cast when canonical primitive values are needed. Selection, compatible casts, fitting fill values, comparisons, and integer aggregates operate through the child. Aggregate partial states are converted back to the logical dtype, and arithmetic keeps the logical overflow behavior through canonical fallback.This draft leaves compressor selection and decimal integration for follow-up work.
#10313 discusses whether dtype and range shortcuts may skip value-dependent errors from lazy children.