Skip to content

Commit 27c6b0d

Browse files
authored
Size the list layout split grid by row count (#10531)
## Summary `ListReader` places scan splits on a grid of outer rows, and the grid was fixed at 64 cells whatever the file size. The grid now grows with the row count: at least 64 cells, and enough that a cell covers at most about 8,192 rows. Files under about 512K rows are unchanged. ## How list splits work `register_splits` must report split points in outer rows, but a list's natural boundaries are its element blocks, which are cut in element rows. It can't read the offsets at that point, so it estimates: 1. Take each element-block boundary as a fraction of all elements. 2. Assume the same fraction of outer rows (i.e. that every list has the same length). 3. Snap it to the nearest edge of a grid of equal cells, and drop duplicates. Example: 800 outer rows, 4,000 elements in 5 blocks, an 8-cell grid: ``` element rows 0 800 1600 2400 3200 4000 |-blk 0-|-blk 1-|-blk 2-|-blk 3-|-blk 4-| boundary at 20% 40% 60% 80% outer rows 0 100 200 300 400 500 600 700 800 |----|----|----|----|----|----|----|----| 8 equal cells snapped to ▲ ▲ ▲ ▲ cell edge 2 3 5 6 (1.6→2, 3.2→3, 4.8→5, 6.4→6) scan splits |=========|====|=========|====|=========| 5 splits 0 200 300 500 600 800 ``` - The grid caps how many splits a list can add (at most one per cell); a list with a single element block adds none. - Grid sizes are powers of two, so the edges of different lists' grids coincide when the scan merges split points across columns. - Known limitation: the estimate assumes uniform list lengths, so with skewed lengths splits have equal rows but unequal work. Neither benchmark dataset is skewed. **Grid size** (this PR): `cells = max(64, next_power_of_two(ceil(rows / 8192)))` | rows | cells | rows per cell | |---:|---:|---:| | 100,000 (statpopgen) | 64 (unchanged) | ~1,560 | | 524,288 | 64 | 8,192 | | 1,000,000 (polarsignals) | 128 (was 64) | ~7,810 | | 10,000,000 | 2,048 | ~4,880 | ## Results Local runs, list layout ÷ develop (52a1d15): 3 interleaved rounds with rotating order, in-process iterations, Apple M5 Max (18 cores), warm page cache, `release_debug` with `target-cpu=native`. "grid 64" is the same list layout without this PR's reader change. | benchmark | grid 64 | this PR | worst query (this PR) | |---|---:|---:|---| | polarsignals, DataFusion, `vortex` | 0.94x | **0.90x** | q4 1.06x (3.7 ms) | | statpopgen, DataFusion, `vortex` | 0.64x | **0.63x** | q0 0.95x | | statpopgen, DuckDB, `vortex` | 0.73x | **0.74x** | q0 0.98x | | statpopgen, DataFusion, `vortex-compact` | | **0.52x** | q0 0.94x | | statpopgen, DuckDB, `vortex-compact` | | **0.61x** | q0 1.01x | <details> <summary>polarsignals, DataFusion, `vortex` (20 iterations)</summary> | query | develop | grid 64 | this PR | grid 64 ÷ develop | this PR ÷ develop | |---|---:|---:|---:|---:|---:| | q0 | 30.4 ms | 45.5 ms | 29.7 ms | 1.49x | 0.98x | | q1 | 109.3 ms | 115.6 ms | 110.7 ms | 1.06x | 1.01x | | q2 | 22.0 ms | 6.5 ms | 7.1 ms | 0.30x | 0.32x | | q3 | 109.8 ms | 117.0 ms | 112.4 ms | 1.07x | 1.02x | | q4 | 3.7 ms | 3.9 ms | 4.0 ms | 1.05x | 1.06x | | q5 | 4.7 ms | 4.7 ms | 4.8 ms | 0.99x | 1.01x | | q6 | 10.9 ms | 10.6 ms | 10.6 ms | 0.97x | 0.98x | | q7 | 4.5 ms | 4.4 ms | 4.4 ms | 0.98x | 0.98x | | q8 | 190.0 ms | 201.6 ms | 191.5 ms | 1.06x | 1.01x | | q9 | 3.5 ms | 3.5 ms | 3.6 ms | 1.00x | 1.02x | | **geomean** | | | | 0.94x | **0.90x** | </details> <details> <summary>statpopgen, DataFusion, `vortex` (5 iterations)</summary> | query | develop | grid 64 | this PR | grid 64 ÷ develop | this PR ÷ develop | |---|---:|---:|---:|---:|---:| | q0 | 0.9 ms | 0.9 ms | 0.9 ms | 0.94x | 0.95x | | q1 | 26.0 ms | 2.1 ms | 1.9 ms | 0.08x | 0.07x | | q2 | 68.1 ms | 46.5 ms | 46.3 ms | 0.68x | 0.68x | | q3 | 244.0 ms | 185.0 ms | 191.4 ms | 0.76x | 0.78x | | q4 | 230.5 ms | 187.1 ms | 180.0 ms | 0.81x | 0.78x | | q5 | 152.4 ms | 117.9 ms | 114.6 ms | 0.77x | 0.75x | | q6 | 506.5 ms | 389.9 ms | 385.0 ms | 0.77x | 0.76x | | q7 | 182.1 ms | 145.6 ms | 146.3 ms | 0.80x | 0.80x | | q8 | 178.9 ms | 145.2 ms | 146.2 ms | 0.81x | 0.82x | | q9 | 173.6 ms | 112.5 ms | 112.8 ms | 0.65x | 0.65x | | q10 | 419.7 ms | 373.5 ms | 364.6 ms | 0.89x | 0.87x | | **geomean** | | | | 0.64x | **0.63x** | </details> <details> <summary>statpopgen, DuckDB, `vortex` (5 iterations)</summary> | query | develop | grid 64 | this PR | grid 64 ÷ develop | this PR ÷ develop | |---|---:|---:|---:|---:|---:| | q0 | 4.8 ms | 4.7 ms | 4.7 ms | 0.98x | 0.98x | | q1 | 5.3 ms | 4.9 ms | 4.8 ms | 0.91x | 0.90x | | q2 | 178.8 ms | 99.0 ms | 101.9 ms | 0.55x | 0.57x | | q3 | 426.4 ms | 325.1 ms | 331.8 ms | 0.76x | 0.78x | | q4 | 424.7 ms | 324.5 ms | 320.4 ms | 0.76x | 0.75x | | q5 | 199.2 ms | 136.4 ms | 133.3 ms | 0.68x | 0.67x | | q6 | 699.2 ms | 473.0 ms | 467.4 ms | 0.68x | 0.67x | | q7 | 85.3 ms | 54.8 ms | 61.2 ms | 0.64x | 0.72x | | q8 | 91.1 ms | 55.6 ms | 53.3 ms | 0.61x | 0.58x | | q9 | 292.0 ms | 233.9 ms | 229.9 ms | 0.80x | 0.79x | | q10 | 1060.5 ms | 852.1 ms | 852.6 ms | 0.80x | 0.80x | | **geomean** | | | | 0.73x | **0.74x** | </details> <details> <summary>statpopgen, DataFusion, `vortex-compact` (5 iterations)</summary> | query | develop | this PR | this PR ÷ develop | |---|---:|---:|---:| | q0 | 0.9 ms | 0.9 ms | 0.94x | | q1 | 44.0 ms | 2.2 ms | 0.05x | | q2 | 91.7 ms | 58.0 ms | 0.63x | | q3 | 254.5 ms | 191.5 ms | 0.75x | | q4 | 253.2 ms | 197.5 ms | 0.78x | | q5 | 182.4 ms | 133.3 ms | 0.73x | | q6 | 521.9 ms | 403.3 ms | 0.77x | | q7 | 521.5 ms | 181.5 ms | 0.35x | | q8 | 521.7 ms | 184.9 ms | 0.35x | | q9 | 171.8 ms | 126.7 ms | 0.74x | | q10 | 444.3 ms | 370.4 ms | 0.83x | | **geomean** | | | **0.52x** | </details> <details> <summary>statpopgen, DuckDB, `vortex-compact` (5 iterations)</summary> | query | develop | this PR | this PR ÷ develop | |---|---:|---:|---:| | q0 | 4.7 ms | 4.8 ms | 1.01x | | q1 | 5.0 ms | 4.9 ms | 0.98x | | q2 | 191.4 ms | 120.4 ms | 0.63x | | q3 | 466.7 ms | 344.9 ms | 0.74x | | q4 | 458.2 ms | 341.1 ms | 0.74x | | q5 | 232.2 ms | 152.9 ms | 0.66x | | q6 | 699.4 ms | 485.3 ms | 0.69x | | q7 | 446.6 ms | 95.0 ms | 0.21x | | q8 | 435.9 ms | 101.5 ms | 0.23x | | q9 | 341.9 ms | 254.6 ms | 0.74x | | q10 | 1090.8 ms | 876.1 ms | 0.80x | | **geomean** | | | **0.61x** | </details> **File sizes** (list layout ÷ develop): statpopgen `vortex` 1.93 → 1.74 GB (0.90x), statpopgen `vortex-compact` 1.003 → 0.977 GB (0.97x), polarsignals 718.8 → 714.2 MB (0.99x). Signed-off-by: Matt Katz <mhkatz97@gmail.com>
1 parent d3bac42 commit 27c6b0d

1 file changed

Lines changed: 30 additions & 6 deletions

File tree

‎vortex-layout/src/layouts/list/reader.rs‎

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,20 @@ type OptionalArrayFuture = BoxFuture<'static, VortexResult<Option<ArrayRef>>>;
4747
/// and above which we evaluate the expression over all rows and intersect afterward.
4848
const EXPR_EVAL_THRESHOLD: f64 = 0.2;
4949

50-
/// Maximum number of outer-row scan ranges contributed by one list layout.
51-
const MAX_LIST_SPLIT_COUNT: u64 = 64;
50+
/// Minimum size of the outer-row split grid, so lists in small files still fan out.
51+
const MIN_LIST_SPLIT_COUNT: u64 = 64;
52+
53+
/// Outer rows per grid cell that the grid grows to stay under for large files.
54+
const TARGET_LIST_SPLIT_ROWS: u64 = 8192;
55+
56+
/// Number of cells in the outer-row split grid: at least [`MIN_LIST_SPLIT_COUNT`], and enough
57+
/// that a cell covers at most about [`TARGET_LIST_SPLIT_ROWS`] outer rows.
58+
fn list_split_grid_size(outer_row_count: u64) -> u64 {
59+
outer_row_count
60+
.div_ceil(TARGET_LIST_SPLIT_ROWS)
61+
.next_power_of_two()
62+
.max(MIN_LIST_SPLIT_COUNT)
63+
}
5264

5365
/// Reader for [`ListLayout`].
5466
#[derive(Clone)]
@@ -376,13 +388,14 @@ impl LayoutReader for ListReader {
376388
)?;
377389

378390
let row_range = split_range.row_range();
391+
let grid_size = list_split_grid_size(self.layout.row_count());
379392
let mut last_split = None;
380393
for element_split in element_splits.into_sorted_deduped() {
381394
let Some(split) = map_element_split_to_outer_grid(
382395
element_split,
383396
element_row_count,
384397
self.layout.row_count(),
385-
MAX_LIST_SPLIT_COUNT,
398+
grid_size,
386399
) else {
387400
continue;
388401
};
@@ -1048,17 +1061,28 @@ mod tests {
10481061

10491062
let mut splits = (1..1_000)
10501063
.filter_map(|split| {
1051-
map_element_split_to_outer_grid(split, 1_000, 100_000, MAX_LIST_SPLIT_COUNT)
1064+
map_element_split_to_outer_grid(split, 1_000, 100_000, MIN_LIST_SPLIT_COUNT)
10521065
})
10531066
.collect::<Vec<_>>();
10541067
splits.dedup();
10551068

1056-
let expected = (1..MAX_LIST_SPLIT_COUNT)
1057-
.map(|grid_index| grid_index * 100_000 / MAX_LIST_SPLIT_COUNT)
1069+
let expected = (1..MIN_LIST_SPLIT_COUNT)
1070+
.map(|grid_index| grid_index * 100_000 / MIN_LIST_SPLIT_COUNT)
10581071
.collect::<Vec<_>>();
10591072
assert_eq!(splits, expected);
10601073
}
10611074

1075+
#[rstest]
1076+
#[case::empty(0, 64)]
1077+
#[case::small(100_000, 64)]
1078+
#[case::floor_boundary(64 * 8192, 64)]
1079+
#[case::past_floor(64 * 8192 + 1, 128)]
1080+
#[case::million(1_000_000, 128)]
1081+
#[case::ten_million(10_000_000, 2048)]
1082+
fn split_grid_grows_with_row_count(#[case] outer_row_count: u64, #[case] expected: u64) {
1083+
assert_eq!(list_split_grid_size(outer_row_count), expected);
1084+
}
1085+
10621086
#[tokio::test]
10631087
async fn nested_list_propagates_element_splits() -> VortexResult<()> {
10641088
let inner = ListArray::try_new(

0 commit comments

Comments
 (0)