ARC WhiteBox Estimation Challenge 2026 - Starter Kit

🎬 60-Second Overview

You are given a randomly-initialized ReLU MLP and a FLOP budget. Predict the per-neuron mean activation under N(0, 1) input — without running anywhere near the budget's worth of forward passes. Your score is that error (MSE against the ground-truth Monte-Carlo means) scaled by the share of the FLOP budget you actually spend — so both accuracy and low compute count. Lower is better.

_{Per-neuron mean activations of a small MLP (width 4, depth 5) after Monte-Carlo ground truth — exactly what your estimator predicts. Generate your own at the hosted WhestBench Explorer.}

The kit is structured as a five-stage ladder of formality: each stage adds one more layer of harness rigor. Start at Stage 1 (pure local math, zero CLI knowledge); climb to Stage 5 (a packaged submission) when you're ready.

🚀 Your First 5 Minutes (Stage 1: just python)

git clone https://github.com/AIcrowd/whest-starterkit.git
cd whest-starterkit

uv sync && uv run python estimator.py

That run printed a Monte-Carlo convergence table — your estimator's predictions, the FLOPs it used, and the MSE against ground truth at increasing sample counts. To experiment, edit predict() in estimator.py and re-run.

Compare against a bundled baseline:

uv run python estimator.py --baseline mean_propagation

🧪 Try the Examples (still Stage 1)

uv run python examples/02_mean_propagation.py
uv run python examples/03_covariance_propagation.py
uv run python examples/04_shipped_weights.py

See examples/README.md for the curriculum table.

🪜 Climb the Ladder (Stages 2-5)

Each rung adds one more layer of harness rigor — climb when you're ready. Per-stage walkthroughs live in the Tutorial.

Stage 1 — Iterate locally · the math: estimator vs Monte Carlo.

uv run python estimator.py

Stage 2 — Validate the contract · contract correctness (shapes, types).

uv run whest validate --estimator estimator.py

Stage 3 — Run on the public set · real scoring against the public Mini split (100 MLPs), in-process, debuggable with pdb.

uv run whest run --estimator estimator.py \
    --dataset hf://aicrowd/arc-whestbench-public-2026@v1-phase1 \
    --split mini \
    --runner local

Stage 4 — Subprocess runner · isolation; closer to the grader environment.

uv run whest run --estimator estimator.py \
    --dataset hf://aicrowd/arc-whestbench-public-2026@v1-phase1 \
    --split mini \
    --runner subprocess

Stage 5 — Package and Submit · build the submission artifact, then ship it (run whest login once first — see Submit to AIcrowd below).

uv run whest package --estimator estimator.py   # build & inspect the tarball
uv run whest submit  --estimator estimator.py   # ship it (also packages, in one step)

These ship only estimator.py — the common case. Embedding weights or splitting across modules? Point --estimator at a folder instead: see Stage 5 → Embedding weights or multiple modules.

🏁 Submit to AIcrowd

Climbed to Stage 5? Ship it from the CLI. Log in once with your AIcrowd API key:

uv run whest login

Then package + submit in one step (add --watch to follow it to a score):

uv run whest submit --estimator estimator.py

Your score and per-MLP detail land on the challenge leaderboard. Full walkthrough: Stage 5 → Submit to AIcrowd.

🚑 When Something Breaks

uv run whest doctor

Reads as a 6-row health check; see docs/reference/whest-doctor.md for what each row means and how to fix warnings.

Or check docs/troubleshooting/.

📚 Documentation

Past Stage 1, the documentation is organized into six sections — pick whichever matches your task. Full map and guided reading paths at docs/.

🪜 Tutorial — Climb the 5-stage ladder above

• Stage 1: Iterate locally — The math; flopscope + local_engine.py, no whest CLI.
• Stage 2: Validate the contract — Class resolves, setup() runs, shape, finite values.
• Stage 3: Run locally — Real scoring against the grader's MLP suite, in-process.
• Stage 4: Subprocess runner — Catches state-bleed, RNG re-use, dirty imports.
• Stage 5: Package and Submit — Build the AIcrowd submission tarball and ship it.

📖 Concepts — Why this challenge exists, what's measured, how ground truth works

• Problem Setup — MLP architecture, He init, the research question, further reading.
• Scoring Model — Pipeline diagram, adjusted_final_layer_score / all_layers_mse formulas, calibration table.
• Ground Truth — How the evaluator computes reference values via Monte Carlo.

🔧 How-to — Recipes: write, debug, optimize, submit

Writing and iterating

• Write an Estimator — Minimal structure, contract checklist, common first failure.
• Inspect MLP Structure — Traversing the MLP object.
• Validate, Run, Package — The standard local loop, plus a useful-flags table.
• Use Evaluation Datasets — Pre-create datasets for fast, reproducible iteration.

Optimizing

• Algorithm Ideas — Monte Carlo, mean propagation, covariance, hybrid, plus open directions.
• Manage FLOP Budget — Where your FLOPs go; line-by-line walkthrough of examples/02.
• Performance Tips — Matmul placement, free ops, env-var knobs.

Debugging and shipping

• Debugging Checklist — Tiered procedure when something feels wrong.
• Pre-Submission Checklist — One-screen gate before you click submit.

📚 Reference — Exact contracts, schemas, lookup material

Estimator API

• Estimator Contract — predict/setup/teardown signatures, SetupContext, failure-semantics table, lifecycle diagram.
• Code Patterns — flopscope patterns, ReLU expectation derivation, when the Gaussian assumption breaks.
• local_engine API — Stage 1's MLP factory and Monte-Carlo helpers.

FLOP and scoring details

• Flopscope Primer — BudgetContext ownership, attribute reference, op cost table.
• Score Report Fields — Every field you'll see in whest run output.

CLI

• CLI Reference — Pointer at the upstream whest CLI.
• whest doctor — The 6 install/env checks and how to fix WARN/FAIL rows.

🚑 Troubleshooting — When something breaks

• Common Participant Errors — Symptom → cause → fix-now → verify.
• FAQ — Quick answers; includes "local score great, submission 10x worse".

🔬 Advanced — Deeper tooling

• Profile Simulation — FLOP and time breakdown of your predict() call.
• WhestBench Explorer — Hosted interactive visualizer at aicrowd.github.io/whestbench-explorer for inspecting MLPs and ground truth.

📁 Repo Layout

├── estimator.py     ← The participant's entry point; every stage operates on this file.
├── local_engine.py  ← Single-file re-implementation of the harness; safe to read end-to-end.
├── examples/        ← Numbered reference estimators (01–03) with a curriculum table.
├── docs/            ← Full documentation; start at docs/README.md.
└── tests/           ← Drift gates: README commands + local_engine parity.

⚖️ License & Contributing

Released under the MIT License. See docs/RELEASING.md for the release process.