Repository navigation
Commit 27c6b0d
authored
Size the list layout split grid by row count (#10531)
## Summary
`ListReader` places scan splits on a grid of outer rows, and the grid
was fixed at 64 cells whatever the file size. The grid now grows with
the row count: at least 64 cells, and enough that a cell covers at most
about 8,192 rows. Files under about 512K rows are unchanged.
## How list splits work
`register_splits` must report split points in outer rows, but a list's
natural boundaries are its element blocks, which are cut in element
rows. It can't read the offsets at that point, so it estimates:
1. Take each element-block boundary as a fraction of all elements.
2. Assume the same fraction of outer rows (i.e. that every list has the
same length).
3. Snap it to the nearest edge of a grid of equal cells, and drop
duplicates.
Example: 800 outer rows, 4,000 elements in 5 blocks, an 8-cell grid:
```
element rows 0 800 1600 2400 3200 4000
|-blk 0-|-blk 1-|-blk 2-|-blk 3-|-blk 4-|
boundary at 20% 40% 60% 80%
outer rows 0 100 200 300 400 500 600 700 800
|----|----|----|----|----|----|----|----| 8 equal cells
snapped to ▲ ▲ ▲ ▲
cell edge 2 3 5 6 (1.6→2, 3.2→3, 4.8→5, 6.4→6)
scan splits |=========|====|=========|====|=========| 5 splits
0 200 300 500 600 800
```
- The grid caps how many splits a list can add (at most one per cell); a
list with a single element block adds none.
- Grid sizes are powers of two, so the edges of different lists' grids
coincide when the scan merges split points across columns.
- Known limitation: the estimate assumes uniform list lengths, so with
skewed lengths splits have equal rows but unequal work. Neither
benchmark dataset is skewed.
**Grid size** (this PR): `cells = max(64, next_power_of_two(ceil(rows /
8192)))`
| rows | cells | rows per cell |
|---:|---:|---:|
| 100,000 (statpopgen) | 64 (unchanged) | ~1,560 |
| 524,288 | 64 | 8,192 |
| 1,000,000 (polarsignals) | 128 (was 64) | ~7,810 |
| 10,000,000 | 2,048 | ~4,880 |
## Results
Local runs, list layout ÷ develop (52a1d15): 3 interleaved rounds
with rotating order, in-process iterations, Apple M5 Max (18 cores),
warm page cache, `release_debug` with `target-cpu=native`. "grid 64" is
the same list layout without this PR's reader change.
| benchmark | grid 64 | this PR | worst query (this PR) |
|---|---:|---:|---|
| polarsignals, DataFusion, `vortex` | 0.94x | **0.90x** | q4 1.06x (3.7
ms) |
| statpopgen, DataFusion, `vortex` | 0.64x | **0.63x** | q0 0.95x |
| statpopgen, DuckDB, `vortex` | 0.73x | **0.74x** | q0 0.98x |
| statpopgen, DataFusion, `vortex-compact` | | **0.52x** | q0 0.94x |
| statpopgen, DuckDB, `vortex-compact` | | **0.61x** | q0 1.01x |
<details>
<summary>polarsignals, DataFusion, `vortex` (20 iterations)</summary>
| query | develop | grid 64 | this PR | grid 64 ÷ develop | this PR ÷
develop |
|---|---:|---:|---:|---:|---:|
| q0 | 30.4 ms | 45.5 ms | 29.7 ms | 1.49x | 0.98x |
| q1 | 109.3 ms | 115.6 ms | 110.7 ms | 1.06x | 1.01x |
| q2 | 22.0 ms | 6.5 ms | 7.1 ms | 0.30x | 0.32x |
| q3 | 109.8 ms | 117.0 ms | 112.4 ms | 1.07x | 1.02x |
| q4 | 3.7 ms | 3.9 ms | 4.0 ms | 1.05x | 1.06x |
| q5 | 4.7 ms | 4.7 ms | 4.8 ms | 0.99x | 1.01x |
| q6 | 10.9 ms | 10.6 ms | 10.6 ms | 0.97x | 0.98x |
| q7 | 4.5 ms | 4.4 ms | 4.4 ms | 0.98x | 0.98x |
| q8 | 190.0 ms | 201.6 ms | 191.5 ms | 1.06x | 1.01x |
| q9 | 3.5 ms | 3.5 ms | 3.6 ms | 1.00x | 1.02x |
| **geomean** | | | | 0.94x | **0.90x** |
</details>
<details>
<summary>statpopgen, DataFusion, `vortex` (5 iterations)</summary>
| query | develop | grid 64 | this PR | grid 64 ÷ develop | this PR ÷
develop |
|---|---:|---:|---:|---:|---:|
| q0 | 0.9 ms | 0.9 ms | 0.9 ms | 0.94x | 0.95x |
| q1 | 26.0 ms | 2.1 ms | 1.9 ms | 0.08x | 0.07x |
| q2 | 68.1 ms | 46.5 ms | 46.3 ms | 0.68x | 0.68x |
| q3 | 244.0 ms | 185.0 ms | 191.4 ms | 0.76x | 0.78x |
| q4 | 230.5 ms | 187.1 ms | 180.0 ms | 0.81x | 0.78x |
| q5 | 152.4 ms | 117.9 ms | 114.6 ms | 0.77x | 0.75x |
| q6 | 506.5 ms | 389.9 ms | 385.0 ms | 0.77x | 0.76x |
| q7 | 182.1 ms | 145.6 ms | 146.3 ms | 0.80x | 0.80x |
| q8 | 178.9 ms | 145.2 ms | 146.2 ms | 0.81x | 0.82x |
| q9 | 173.6 ms | 112.5 ms | 112.8 ms | 0.65x | 0.65x |
| q10 | 419.7 ms | 373.5 ms | 364.6 ms | 0.89x | 0.87x |
| **geomean** | | | | 0.64x | **0.63x** |
</details>
<details>
<summary>statpopgen, DuckDB, `vortex` (5 iterations)</summary>
| query | develop | grid 64 | this PR | grid 64 ÷ develop | this PR ÷
develop |
|---|---:|---:|---:|---:|---:|
| q0 | 4.8 ms | 4.7 ms | 4.7 ms | 0.98x | 0.98x |
| q1 | 5.3 ms | 4.9 ms | 4.8 ms | 0.91x | 0.90x |
| q2 | 178.8 ms | 99.0 ms | 101.9 ms | 0.55x | 0.57x |
| q3 | 426.4 ms | 325.1 ms | 331.8 ms | 0.76x | 0.78x |
| q4 | 424.7 ms | 324.5 ms | 320.4 ms | 0.76x | 0.75x |
| q5 | 199.2 ms | 136.4 ms | 133.3 ms | 0.68x | 0.67x |
| q6 | 699.2 ms | 473.0 ms | 467.4 ms | 0.68x | 0.67x |
| q7 | 85.3 ms | 54.8 ms | 61.2 ms | 0.64x | 0.72x |
| q8 | 91.1 ms | 55.6 ms | 53.3 ms | 0.61x | 0.58x |
| q9 | 292.0 ms | 233.9 ms | 229.9 ms | 0.80x | 0.79x |
| q10 | 1060.5 ms | 852.1 ms | 852.6 ms | 0.80x | 0.80x |
| **geomean** | | | | 0.73x | **0.74x** |
</details>
<details>
<summary>statpopgen, DataFusion, `vortex-compact` (5
iterations)</summary>
| query | develop | this PR | this PR ÷ develop |
|---|---:|---:|---:|
| q0 | 0.9 ms | 0.9 ms | 0.94x |
| q1 | 44.0 ms | 2.2 ms | 0.05x |
| q2 | 91.7 ms | 58.0 ms | 0.63x |
| q3 | 254.5 ms | 191.5 ms | 0.75x |
| q4 | 253.2 ms | 197.5 ms | 0.78x |
| q5 | 182.4 ms | 133.3 ms | 0.73x |
| q6 | 521.9 ms | 403.3 ms | 0.77x |
| q7 | 521.5 ms | 181.5 ms | 0.35x |
| q8 | 521.7 ms | 184.9 ms | 0.35x |
| q9 | 171.8 ms | 126.7 ms | 0.74x |
| q10 | 444.3 ms | 370.4 ms | 0.83x |
| **geomean** | | | **0.52x** |
</details>
<details>
<summary>statpopgen, DuckDB, `vortex-compact` (5 iterations)</summary>
| query | develop | this PR | this PR ÷ develop |
|---|---:|---:|---:|
| q0 | 4.7 ms | 4.8 ms | 1.01x |
| q1 | 5.0 ms | 4.9 ms | 0.98x |
| q2 | 191.4 ms | 120.4 ms | 0.63x |
| q3 | 466.7 ms | 344.9 ms | 0.74x |
| q4 | 458.2 ms | 341.1 ms | 0.74x |
| q5 | 232.2 ms | 152.9 ms | 0.66x |
| q6 | 699.4 ms | 485.3 ms | 0.69x |
| q7 | 446.6 ms | 95.0 ms | 0.21x |
| q8 | 435.9 ms | 101.5 ms | 0.23x |
| q9 | 341.9 ms | 254.6 ms | 0.74x |
| q10 | 1090.8 ms | 876.1 ms | 0.80x |
| **geomean** | | | **0.61x** |
</details>
**File sizes** (list layout ÷ develop): statpopgen `vortex` 1.93 → 1.74
GB (0.90x), statpopgen `vortex-compact` 1.003 → 0.977 GB (0.97x),
polarsignals 718.8 → 714.2 MB (0.99x).
Signed-off-by: Matt Katz <mhkatz97@gmail.com>1 parent d3bac42 commit 27c6b0d
1 file changed
Lines changed: 30 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
51 | | - | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
52 | 64 | | |
53 | 65 | | |
54 | 66 | | |
| |||
376 | 388 | | |
377 | 389 | | |
378 | 390 | | |
| 391 | + | |
379 | 392 | | |
380 | 393 | | |
381 | 394 | | |
382 | 395 | | |
383 | 396 | | |
384 | 397 | | |
385 | | - | |
| 398 | + | |
386 | 399 | | |
387 | 400 | | |
388 | 401 | | |
| |||
1048 | 1061 | | |
1049 | 1062 | | |
1050 | 1063 | | |
1051 | | - | |
| 1064 | + | |
1052 | 1065 | | |
1053 | 1066 | | |
1054 | 1067 | | |
1055 | 1068 | | |
1056 | | - | |
1057 | | - | |
| 1069 | + | |
| 1070 | + | |
1058 | 1071 | | |
1059 | 1072 | | |
1060 | 1073 | | |
1061 | 1074 | | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
1062 | 1086 | | |
1063 | 1087 | | |
1064 | 1088 | | |
| |||
0 commit comments