Add Westermo node_exporter metrics benchmark

[joseph-isaacs]

Summary

• Tracking Issue: Epic: Time series data support #10171

Builds on #10213, now merged into develop. This is the first of two PRs that add benchmarks for Prometheus-style metrics data. It adds a westermo vx-bench suite over real node_exporter data, which runs in CI only under action/bench-sql-extended. The next PR adds lo2, which runs real PromQL queries.

The data is the Westermo test system performance data set, licensed CC BY 4.0. It is node_exporter scrapes from 19 servers every 30 seconds for a month, pinned to upstream commit 47e0ccdc, giving 434 series and about 37.5 million samples. The 15 queries are PromQL-shaped expressions written for this suite, each translated to SQL with the PromQL in a comment above it. They are not taken from a real workload; the lo2 PR covers that.

Changes

• Data conversion. vortex-bench/src/westermo/ downloads the 19 CSVs and converts them to Prometheus layout. The parser rejects both missing and extra CSV values, including trailing commas, with regression cases for row width:
  • one row per sample, with a labels struct (__name__, instance, job), an int64 millisecond ts and a float64 value;
  • rows sorted by series, then time.
• Queries and docs. vortex-bench/sql/westermo.sql holds the queries and westermo.md documents the suite. The queries cover:
  • single-series range reads;
  • aggregation by label at a fixed step;
  • regex and negated regex matchers;
  • a one-to-one join;
  • newest-point and newest-N queries;
  • the label values, metric names and series metadata APIs.
• Portable SQL. Every query runs unchanged on DataFusion and DuckDB: time bounds are integer literals, steps are ts - ts % step, and regexes are anchored.
• CI. A catalog entry schedules the suite in the pr-full preset only, on DataFusion and DuckDB over Parquet and Vortex. It does not run on develop, pr, pr-compact, pr-all or nightly. The matrix test and docs/developer-guide/benchmarking.md are updated to match.
• Registration. The suite is registered in the benchmark CLI, the dataset enum, the v3 emitter, the orchestrator config and README, and the bench-performance skill.

Local results on a 4-core cloud VM, 5 iterations per query, with files in the page cache:

	DataFusion	DuckDB
Vortex time relative to Parquet, geometric mean	0.41	0.55
Queries where Vortex is faster	14 of 15	13 of 15

File	Size
Parquet, zstd level 3	123 MB
Vortex	219 MB

Validation before the October 5 rebase:

• uv run --project bench-orchestrator --with pytest pytest bench-orchestrator/tests/test_matrix.py: 13 passed.
• uvx ruff format --check and uvx ruff check on bench-orchestrator: clean.
• vx-bench matrix pr-full lists westermo; the other presets do not.
• Before rebasing onto Split SQL PR benchmarks into bench-sql and bench-sql-extended #10213: datafusion-bench, duckdb-bench and data-gen built in release_debug with no warnings, and vx-bench run westermo -e datafusion,duckdb -f parquet,vortex -i 5 completed all 60 runs.
• All 15 queries return the same rows on DataFusion and DuckDB, checked on a Python copy of the conversion. The only differences are last-digit float differences from summation order.

Current revision:

• Rebased the Westermo commits onto current develop after Split SQL PR benchmarks into bench-sql and bench-sql-extended #10213 merged, removing the parent PR commits from this diff.
• Fixed CSV row-width validation and added regression cases for short rows, extra values, and trailing commas.
• Added an explicit CI matrix assertion for the four Westermo engine/format targets.
• Added missing SPDX headers and constructor documentation.
• No local tests, linting, formatting, builds, or benchmarks were run for this update, following the repository guidance that verification is user-selected. The new regression cases and rebased code await CI.

The old head's only failed check was CodSpeed Performance Analysis. The preceding revision passed that gate, and the intervening commit only formatted westermo/benchmark.rs. The report also warned of different runtime environments for wall-time measurements. This appears unrelated to the Westermo changes; the updated branch will receive fresh CI results.

[codspeed]

Merging this PR will regress 4 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 4 regressed benchmarks
✅ 2099 untouched benchmarks
⏩ 503 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	mul_u64_nonnull_neon	28.8 µs	39.5 µs	-27.18%
❌	WallTime	mul_i64_nonnull_neon	32.7 µs	38.8 µs	-15.88%
❌	WallTime	multiply_shapes_neon[(32768, PerRowPerRow)]	32.6 µs	38.5 µs	-15.32%
❌	WallTime	dict_canonicalize_gt_u8_neon[1000000]	492 µs	547.8 µs	-10.19%
⚡	Simulation	once[dict]	1,118.5 µs	492.2 µs	×2.3
⚡	Simulation	repeated[dict]	784.8 µs	521.6 µs	+50.47%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ji/westermo-bench (c38bf58) with develop (6135e5e)²}

Footnotes

1. 503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on develop (2cc04c4) during the generation of this report, so 6135e5e was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[joseph-isaacs]

CodSpeed Performance Analysis fails on b4c33cf with regressions in decode_bitpacked_chunked_{avx512,avx2,neon}[i64, 524288], decompress[alp_for_bp_f64] and decompress[datetime_for_bp]. These do not come from this PR:

• This PR changes only vortex-bench (new westermo suite), bench-orchestrator and docs. It touches nothing in encodings/fastlanes, vortex or the decompress paths those benchmarks measure.
• The previous run on 6f30a08, the same diff before the format-only commit b4c33cf, reported no regressions in these benchmarks.
• CodSpeed itself warns that the WallTime runs were compared across different runtime environments on hosted runners.

There is no fix to port. I'm leaving the check as is; it can be acknowledged in CodSpeed.

---

Generated by Claude Code