Split SQL PR benchmarks into bench-sql and bench-sql-extended

`action/bench-sql` keeps the quicker `pr` preset, now without Clickbench
Sorted. The new `action/bench-sql-extended` label runs the `pr-full`
preset, which covers every regular SQL benchmark.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFeuTV21uQCX8GcNyZbKwf

Author: claude

.github/workflows/pr-bench-dispatch.yml

@@ -151,3 +151,21 @@ jobs:
     secrets: inherit
     with:
       matrix_preset: "pr-compact"
+
+  remove-sql-extended-label:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    if: github.event.label.name == 'action/bench-sql-extended'
+    steps:
+      - uses: actions-ecosystem/action-remove-labels@2ce5d41b4b6aa8503e285553f75ed56e0a40bae0  # v1
+        if: github.event.pull_request.head.repo.full_name == 'vortex-data/vortex'
+        with:
+          labels: action/bench-sql-extended
+          fail_on_error: true
+
+  sql-extended-bench:
+    needs: remove-sql-extended-label
+    uses: ./.github/workflows/pr-bench-sql.yml
+    secrets: inherit
+    with:
+      matrix_preset: "pr-full"

bench-orchestrator/bench_orchestrator/ci_matrix/catalog.py

@@ -13,10 +13,10 @@
 
 PRESETS = {
     "develop": "Every regular SQL benchmark at full target coverage.",
-    "pr": "The quicker pull-request SQL benchmark matrix.",
+    "pr": "The quicker pull-request SQL benchmark matrix (action/bench-sql).",
     "pr-compact": "Pull-request SQL benchmarks for Vortex Compact plus Parquet controls.",
-    "pr-all": "The union of the focused PR and PR Compact benchmark matrices.",
-    "pr-full": "Every regular SQL benchmark at full PR target coverage.",
+    "pr-all": "The PR and PR Compact benchmark matrices plus Clickbench Sorted (action/bench-all).",
+    "pr-full": "Every regular SQL benchmark at full PR target coverage (action/bench-sql-extended).",
     "nightly": "Large-scale SF=100 TPC-H on NVMe and S3 at default targets.",
 }
 
@@ -98,7 +98,6 @@
         benchmark=Benchmark.CLICKBENCH_SORTED,
         name="Clickbench Sorted on NVME",
         runs={
-            "pr": DEFAULT,
             "pr-compact": COMPACT,
             "pr-all": STANDARD,
             "pr-full": DEFAULT_WITH_DUCKDB_PR_FULL,

bench-orchestrator/tests/test_matrix.py

@@ -42,7 +42,7 @@
     "pr": tuple(
         benchmark_id
         for benchmark_id in REGULAR_IDS
-        if benchmark_id not in {"tpch-s3-10", "appian-nvme", "vortex-queries"}
+        if benchmark_id not in {"clickbench-sorted-nvme", "tpch-s3-10", "appian-nvme", "vortex-queries"}
     ),
     "pr-compact": COMPACT_IDS,
     "pr-all": PR_ALL_IDS,
@@ -92,22 +92,17 @@ def test_pr_target_selection() -> None:
         assert set(cast("list[str]", entry["data_formats"])) == {"parquet", "vortex-compact"}
 
 
-def test_pr_all_is_union_of_focused_presets() -> None:
+def test_pr_all_covers_focused_presets() -> None:
     pr = {entry["id"]: entry for entry in _entries("pr")}
     pr_compact = {entry["id"]: entry for entry in _entries("pr-compact")}
     pr_all = {entry["id"]: entry for entry in _entries("pr-all")}
 
     assert set(pr_all) == set(pr) | set(pr_compact)
     for benchmark_id, entry in pr_all.items():
-        expected_targets: set[tuple[str, str]] = set()
-        expected_formats: set[str] = set()
         for preset in (pr, pr_compact):
             if source := preset.get(benchmark_id):
-                expected_targets |= _targets(source)
-                expected_formats |= set(cast("list[str]", source["data_formats"]))
-
-        assert _targets(entry) == expected_targets
-        assert set(cast("list[str]", entry["data_formats"])) == expected_formats
+                assert _targets(source) <= _targets(entry)
+                assert set(cast("list[str]", source["data_formats"])) <= set(cast("list[str]", entry["data_formats"]))
 
 
 def test_resolver_rejects_empty_targets() -> None:

docs/developer-guide/benchmarking.md

@@ -261,13 +261,16 @@ Benchmarks run automatically on all commits to `develop` and can be run on-deman
 - **String encoding** -- `action/bench-string` runs only the string encoding benchmark.
 - **GPU compression** -- `action/bench-gpu-compress` runs the allow-listed Vortex decompression
   cases on a GPU runner.
-- **SQL** -- `action/bench-sql` runs the `pr` preset, which excludes `vortex-compact`.
+- **SQL** -- `action/bench-sql` runs the `pr` preset, a quick core subset of the SQL benchmarks
+  that excludes `vortex-compact`, Clickbench Sorted, Appian, TPC-H SF=10 on S3, and Vortex queries.
+- **SQL Extended** -- `action/bench-sql-extended` runs the `pr-full` preset: every regular SQL
+  benchmark, including `vortex-compact` and DuckDB-format targets.
 - **SQL Compact** -- `action/bench-sql-compact` runs the `pr-compact` preset, which benchmarks
   `vortex-compact` plus Parquet control rows used to distinguish code changes from runner drift.
 - **All CPU benchmarks** -- `action/bench-all` runs random access, compression, string encoding,
-  and the `pr-all` SQL preset, which combines the `pr` and `pr-compact` coverage without
-  repeating shared jobs. Do not combine it with other benchmark labels; GPU compression is the
-  only exception.
+  and the `pr-all` SQL preset, which combines the `pr` and `pr-compact` coverage, plus
+  Clickbench Sorted, without repeating shared jobs. Do not combine it with other benchmark labels;
+  GPU compression is the only exception.
 
 All CI benchmarks run on dedicated instances with the `release_debug` profile and
 `-C target-cpu=native` to produce representative numbers.