pytest-perfguard

pytest-perfguard records CPU and memory metrics for ordinary pytest tests and compares them with compatible historical results. It does not require benchmark fixtures, decorators, or markers.

This repository currently contains a proof of concept. Its goals are:

• measure the existing test run instead of repeating tests;
• use Linux hardware instruction counters when permitted;
• fall back explicitly to process CPU time when counters are unavailable;
• collect process RSS with no sampling thread or binary dependency;
• keep history in GitLab job artifacts rather than an external service;
• catch both single-test jumps and suite-wide accumulation;
• retain a fixed anchor so gradual drift remains visible after the rolling baseline adapts;
• keep performance regressions non-blocking without masking functional failures.

How it compares

Perfguard is a broad regression smoke detector for an existing test suite. It does not replace a controlled microbenchmark harness or an allocation profiler; those tools trade more setup or runtime for a more precise answer.

Tool	Test changes and run model	Signals and history	Best fit and trade-off
pytest-perfguard	No test changes; measures one normal pytest run	CPU instructions when available, process CPU time, and RSS; rolling and fixed baselines in GitLab artifacts	Broad, non-blocking CI coverage without a service; measurements are coarser than isolated benchmarks or allocation tracing
pytest-benchmark	Uses the benchmark fixture and calibrated repetitions	Wall-clock distributions; JSON or Elasticsearch storage, comparisons, and fail expressions	Precise microbenchmarks; requires benchmark-specific tests and adds repeated execution
pytest-codspeed	Uses a benchmark marker or fixture and instrumented benchmark runs	CPU simulation, wall time, or memory measurements with managed history and a dashboard	Stable hosted comparison workflow; requires selected benchmarks and the CodSpeed service
asv	Uses a dedicated benchmark suite and can run it across commits and environment matrices	Timing, object or peak memory, and arbitrary numeric values; JSON history and a static web UI	Long-term package and interpreter comparisons; more setup, storage, and CI time
pytest-monitor	Automatically records ordinary tests after installation	Duration, CPU usage, and memory in local SQLite or an optional server	The closest annotation-free collector; comparison and alert policy must be built around its recorded data
pytest-memray	Traces existing tests with --memray; memory limits use markers	Allocation and peak-memory reports, per-test limits, and increase checks	Deep memory diagnosis; native allocation tracing costs more and threshold gates are explicit
pytest-leaks	Repeats each test through a command-line option	CPython reference-count and memory-block leaks; no historical baseline	Targeted reference-leak detection; requires a debug Python build and substantially extends the run

In practice, use Perfguard when the first requirement is coverage of unchanged tests with local GitLab history and low additional CI work. Use pytest-benchmark or pyperf for carefully isolated hot paths, CodSpeed when a managed dashboard is desirable, and ASV for performance across many commits, Python versions, or dependency versions. Memray and pytest-leaks are better follow-up tools when a broad RSS signal needs an allocation or reference-leak diagnosis.

Install

Add the package to the same environment as pytest:

uv add --dev pytest-perfguard==0.1.1

The plugin is automatically discovered through pytest's pytest11 entry point. CI enables collection with:

PERFGUARD=1 pytest

It writes one artifact per GitLab test job under .perfguard/current/<CI_JOB_NAME_SLUG>-<CI_JOB_ID>.json, avoiding collisions when GitLab downloads several jobs' artifacts. With pytest-xdist, workers send their results to the controller and the controller writes one job-level file.

Non-blocking GitLab design

Do not put allow_failure: true on the normal test job: that would also make functional test failures non-blocking.

Instead, retain the normal blocking test job and add small report jobs. The default-branch job keeps the rolling database, while merge-request output expires:

.perfguard-report:
  stage: report
  needs:
    - job: test
      artifacts: true
  script:
    - uvx --from "pytest-perfguard==${PERFGUARD_VERSION}" pytest-perfguard gitlab
  when: always
  allow_failure: true
  artifacts:
    when: always
    paths:
      - .perfguard/results.json
      - .perfguard/junit.xml
    reports:
      junit: .perfguard/junit.xml

perfguard:
  extends: .perfguard-report
  artifacts:
    expire_in: never
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: always

perfguard-mr:
  extends: .perfguard-report
  artifacts:
    expire_in: 30 days
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: always

The complete example is in examples/gitlab-ci.yml. Projects with several test jobs list each producer under needs. On a merge request, the command downloads the latest successful target-branch perfguard artifact using CI_JOB_TOKEN. On the default branch it records the current samples into rolling history while preserving the fixed anchor. No GitLab credentials enter the application test container.

The result job returns:

• 0 when no confirmed regression is found, including warm-up and persistent drift;
• 1 when a regression is found;
• 2 when artifact preparation or comparison fails.

Because only this reporting job has allow_failure: true, all three outcomes remain visible without weakening the test gate.

The command prints the largest regressions and persistent drifts directly in the job log and writes a JUnit report. Confirmed regressions are failures; drift is a non-failing skipped testcase. Node IDs, metrics, baselines, and limits therefore appear in GitLab without downloading JSON.

Baseline policy

Results are compared only when architecture, CPU model, measurement backends, xdist worker count, and runner key match. On GitLab the runner key defaults to CI_RUNNER_DESCRIPTION, then CI_RUNNER_ID; PERFGUARD_RUNNER_KEY can group a known-homogeneous pool or separate another CI provider's machines. This prevents different shared-runner hosts from teaching one noisy baseline. Python version, lock hash, and commit SHA are recorded as diagnostic context rather than compatibility keys, so interpreter and dependency upgrades are compared.

Each compatible series keeps two baselines with different purposes:

• a rolling history estimates current noise with median and MAD;
• a fixed anchor is created automatically after three successful default-branch samples and never moves during normal recording.

A measurement that exceeds the rolling limit is a confirmed regression. If it exceeds only the fixed-anchor limit, it is reported as persistent drift without a failing exit code. Requiring rolling corroboration prevents a short, unusually fast anchor from making every later stable run flaky, while the fixed anchor still prevents gradual degradation from disappearing silently as history moves. To intentionally rebase after an understood change, run one successful default-branch pipeline with either:

pytest-perfguard gitlab --accept-baseline

or PERFGUARD_ACCEPT_BASELINE=1. Acceptance is rejected on merge-request and other compare-only pipelines. Do not leave the variable enabled permanently.

The initial thresholds require both a relative increase and an absolute floor:

Metric	Relative increase	Absolute floor
CPU instructions (per test and summed job)	15%	1,000,000 instructions
Process CPU time (summed job and session)	15%	5 ms
RSS boundary growth (summed positive growth and per-test attribution)	20%	16 MiB
Process end and peak RSS	20%	16 MiB

Three compatible historical samples are required before gating. Thresholds and history length can be changed by CLI options.

Job aggregates are compared only when the set of passed test node IDs matches. They catch regressions that are individually below the absolute floor, such as 1,000 tests each becoming 2 ms slower or several tests each retaining 8 MiB. Individual baselines use the source of the test function, so editing a different test in the same file does not reset them.

Per-test process CPU and RSS growth remain in the raw artifact, but become regression attribution only when the corresponding job aggregate also exceeds its rolling limit. RSS attribution is corroborated specifically by summed positive RSS growth; process end and peak RSS remain independent job-level guards. Shared runners can move unrelated process work, garbage collection, lazy initialization, and allocator growth between individual pytest calls; allowing an isolated call spike to fail the report produced false positives in real xdist trials. Per-test instruction counts remain independently guarded.

Measurement scope and limitations

• Linux instruction counting uses perf_event_open. Each pytest process runs one brief startup calibration and validates the counter's scheduled time on every read. Access denial, a zero/nonfunctional counter, no scheduled time, or less than 90% scheduled time causes an automatic process_time fallback; the artifact records the exact unavailable reason. This avoids both false zero-instruction baselines and noisy heavily multiplexed baselines.
• CPU time is process CPU consumed during each pytest call. Hardware instructions count only the thread running that call. The artifact records these scopes explicitly. Summed job and session CPU remain guarded so threaded work is not silently ignored; individual CPU-time outliers are reported only as attribution for a corroborated job regression.
• RSS uses /proc/self/statm on Linux, working set on Windows, and peak RSS as a fallback. With xdist, process RSS is summed across the controller and workers, and per-test CPU is summed across workers; the artifact records the process count.
• RSS growth is a regression signal, not proof of a memory leak. Allocation tracing and repeated workloads remain diagnostic follow-ups.
• The proof of concept measures the pytest process. Separate application services require a service- or cgroup-level collector.
• Shared runners with different CPU models build separate history buckets.
• The default-branch .perfguard/results.json is the baseline database and uses expire_in: never; deleting it restarts warm-up. Merge-request comparison artifacts expire after 30 days in the example.