Skip to content

benchmarking.md

Latest commit

 

History

History
173 lines (137 loc) · 6.85 KB

benchmarking.md

File metadata and controls

173 lines (137 loc) · 6.85 KB
Raw

Benchmarking Tensor Factors

This page explains how to benchmark the tensor factor engine on synthetic data. The benchmark is designed to be reproducible without proprietary market data.

Quick Start

python -m pip install -e '.[dev]'
python scripts/benchmark_tensor_factors.py \
  --device cpu --n-dates 750 --n-stocks 1000 --window 20 \
  --repeat 10 --warmup 3 --threads 1 --interop-threads 1 --seed 42 \
  --json-out artifacts/benchmark-v1.json

This is the canonical cross-platform protocol v1 CPU command. It writes a machine-readable report to artifacts/benchmark-v1.json. On systems with GNU Make, make benchmark is an equivalent shortcut.

python scripts/benchmark_tensor_factors.py \
  --device cpu \
  --n-dates 750 \
  --n-stocks 1000 \
  --window 20 \
  --repeat 10 \
  --warmup 3 \
  --threads 1 \
  --interop-threads 1 \
  --seed 42 \
  --json-out artifacts/benchmark-v1.json

Protocol v1 fixes the panel, seed, rolling window, repetitions, and PyTorch thread pools. Submit this CPU result and the generated JSON before optional variants so reports remain comparable across machines.

Optional GPU Run

Run the same fixed workload on CUDA separately:

python scripts/benchmark_tensor_factors.py --device cuda

GPU results belong in their own comparison group. Do not interpret a CPU/GPU or multi-thread/single-thread difference as a controlled hardware ranking.

Larger Panel

python scripts/benchmark_tensor_factors.py \
  --device auto \
  --n-dates 1500 \
  --n-stocks 3000 \
  --window 20 \
  --repeat 10 \
  --warmup 3 \
  --threads 1 \
  --interop-threads 1 \
  --seed 42

The script prints a Markdown table with environment details, mean runtime, runtime standard deviation, and peak CUDA memory when available.

What It Measures

The benchmark covers representative operations:

  • cs_rank(close)
  • rolling ts_mean
  • rolling ts_rank
  • rolling ts_corr
  • ewma
  • compute_legacy_set on a six-factor subset

The goal is to compare factor-engine behavior across machines and devices, not to claim universal throughput. Results depend on PyTorch version, CPU, GPU, CUDA runtime, memory bandwidth, panel shape, factor subset, and mask density.

Reporting Results

When sharing benchmark results, include:

  • commit SHA
  • command
  • Python version
  • PyTorch version
  • CPU and GPU model
  • CUDA availability, CUDA version, and CUDA device name
  • protocol version, thread counts, seed, panel size, window, repeat, and warmup
  • whether the run used CPU, CUDA, or both

Interpreting Results

Small panels may not benefit from GPU execution because transfer overhead and kernel launch overhead dominate runtime. Larger panels and wider universes are more likely to show GPU advantages, especially for rolling operations and multi-factor computation.

Use this benchmark as an engineering diagnostic. It is separate from backtest quality, factor IC, or trading performance.

Opt-in linear-memory moments

The factor APIs and feature CLI can select centred FP64 block mean/std/ cov/corr on CPU or CUDA. Other primitives retain their existing implementations.

from mlquant.features import compute_legacy_set
factors, mask, names = compute_legacy_set(panel, rolling_backend="linear")
mlquant features --config configs/small.yaml --rolling-backend linear
python scripts/benchmark_factor_backends.py \
  --n-dates 500 --n-stocks 200 --warmup 2 --repeat 5 --seed 42 \
  --json-out artifacts/factor-backends.json

The config key rolling_backend: linear also applies to mlquant demo; an explicit feature CLI option overrides it. features.pt records the selection. Both factor-set APIs default to unfold even inside an outer scope. For direct primitive or registry calls, use with using_rolling_backend("linear"): from mlquant.features; scopes restore on exceptions and remain local to threads/ async contexts.

This is a numerical choice, not a bit-identical acceleration. Linear moments exclude nonfinite observations, clamp correlations and use FP64 accumulation. Constant-window correlations and near-ties can therefore change downstream ranks materially. Disjoint block prefixes/suffixes avoid subtracting cumulative statistics across the entire history, including large past outliers; extreme within-window dynamic ranges still require a direct FP64 reference. MPS lacks FP64 and is rejected. Compare factor values, masks and ranks before changing a research experiment's backend. The full-engine benchmark reports all 213 factors' own masks and value errors, sampled rank differences, and a sampled independent FP64 oracle for correlations of discrete rank inputs. Its fresh-process timings and RSS snapshots precede saving and diagnostics; neither kernel speedups nor this synthetic benchmark establish predictive quality.

Consume one factor at a time

For per-factor screening, IC with your own labels, or incremental storage, avoid retaining the full [T,N,213] output:

import torch
from mlquant.features import iter_legacy_factors
with torch.no_grad():
    for name, values, mask in iter_legacy_factors(panel):
        # Compute/store this factor, then discard it before requesting the next.
        print(name, values.shape)

Values, names and masks follow compute_legacy_set, including optional neutralize, names, and rolling_backend. Backend scopes end before each yield. Keep the panel and registry fixed and do not mutate returned masks. Retaining all outputs or autograd graphs defeats bounded consumption; a trainer requiring every feature dimension still needs an appropriate materialization. Run scripts/benchmark_factor_streaming.py --json-out artifacts/streaming.json for a fresh-process full-213-factor mean/RMS screening comparison.

Daily IC screening

python scripts/eval_factor_ic.py --config configs/small.yaml --json-out artifacts/daily-ic.json streams all 213 factors on a synthetic panel, with limit masking before factor construction. Repeat --factor NAME to select a subset. The API mlquant.backtest.daily_factor_ic(values, targets, factor_mask, label_mask) calculates Pearson IC and average-tie Spearman RankIC across assets on each date. It excludes nonfinite pairs before ranking; undersized or constant cross-sections are missing dates rather than zero IC. FP64 reductions run on CPU/CUDA.

The script uses adjacent, available, finite positive raw closes for labels; future label availability is only evaluation eligibility. It never enters the factor panel's mask. Summaries weight each valid date equally, report pair counts and retain the entire daily series. IC IR is mean / sample standard deviation, unannualized, without correcting temporal dependence. No factor selection or test holdout is performed; full-sample synthetic IC is diagnostic, not alpha. The earlier script pooled stock-days and applied limit masking after factor computation, so old results from that script have different semantics.