English | 简体中文
TomlSmith Benchmark measures end-to-end cold-start latency, throughput, and peak memory for TOML checking and formatting CLIs. Each sample runs the product command in a new process instead of calling a parser library directly.
The check scenario uses deterministic synthetic documents covering the published TOML 1.0.0 and TOML 1.1.0 grammars. Only product/version pairs that accept the document under the selected version are shown.
|
TOML 1.0 ·
|
TOML 1.1 ·
|
The format scenario uses the same documents in an intentionally edited layout. A result is included only when the output reparses under the selected version, preserves decoded content, and is unchanged by a second format pass.
|
TOML 1.0 ·
|
TOML 1.1 ·
|
* The correctness check compares decoded TOML content, not comments or layout. tomll reparses and writes the data, discarding comments and changing literal styles. The other formatters preserve comments, so these rows do not represent identical work.
These are the same four canonical results published on tomlsmith.github.io. Git stores only the generated tables and charts. Raw run directories and archives are intentionally ignored; if they need to be distributed, publish them as checksummed GitHub Release assets rather than source files.
Install the exact published TomlSmith native CLI and the pinned competitor CLIs (prerequisites are in CONTRIBUTING.md):
./scripts/setup-products.shRun any exact lane, for example:
TOMLSMITH_BENCH_FILTER=e2e/check/cold-stdin/1.0/v1_0_medium \
./scripts/run-bench.sh local-check-v1_0
TOMLSMITH_BENCH_FILTER=e2e/check/cold-stdin/1.1/v1_1_medium \
./scripts/run-bench.sh local-check-v1_1
TOMLSMITH_BENCH_FILTER=e2e/format/cold-stdin/1.0/v1_0_medium \
./scripts/run-bench.sh local-format-v1_0
TOMLSMITH_BENCH_FILTER=e2e/format/cold-stdin/1.1/v1_1_medium \
./scripts/run-bench.sh local-format-v1_1The manually dispatched Benchmark workflow accepts a tomlsmith_ref input to build the TomlSmith CLI from any git ref of the core repository before running the selected lanes. A locally built TomlSmith executable can be measured by exporting TOMLSMITH_BIN with TOMLSMITH_BIN_EXPECTED_VERSION=any (or an explicit version string) to relax the exact release pin, and TOMLSMITH_BENCH_SKIP_PEAK_RSS=1 skips the peak-RSS pass on hosts without GNU time; published lanes leave both unset. Every timed sample starts a fresh process and reads the document from stdin; Peak RSS is the median of three separate fresh runs. Summarize any result directory with scripts/summarize-results.py and regenerate the charts with scripts/generate-result-charts.py.
Correctness, preflight, and resource-sampling process trees are bounded by TOMLSMITH_BENCH_PROCESS_TIMEOUT_SECS (120 seconds by default). The cross-product gate covers the four canonical medium-fixture publication lanes, while every other selected lane—including large and stress fixtures—must pass its own bounded preflight before timing. Timed iterations use the direct spawn path after that preflight so containment overhead is not counted as product latency. See the complete measurement methodology.
Run the pull-request gate with:
./scripts/check.shBefore publishing measurements, run the cross-product correctness gate for all four canonical publication lanes with ./scripts/check-products.sh; any additional lane is verified again against its exact fixture by the benchmark runner. Scheduled and on-demand benchmark workflows upload complete result directories as temporary GitHub Actions artifacts. Raw directories and archives stay out of Git; a maintainer can promote an archive and its checksum to a GitHub Release when long-lived public evidence is needed.
Read CONTRIBUTING.md before opening a pull request. Participation is governed by the organization-wide Code of Conduct.
MIT. See LICENSE.