Repository navigation
Goal: be able to evaluate math competition/textbook/past papers #718
Description
Activity
Source survey, and a workflow that can actually reach full coverage
This started from someone asking whether Humanity's Last Exam is a target for
this goal. Short answer: almost none of the famous AI-era math benchmarks are, and the ones that
matter for us are older, duller and far larger. The long answer is a survey, a licensing line we
should not cross, and a loop.Everything below was checked against the source rather than recalled; where a fact could not be
confirmed it says so.
1. The distinction that decides which sources are useful
An LLM math benchmark scores an answer to a natural-language question. Our unit of work is
an expression and a transformation of it, and the thing that makes a corpus valuable to us is not
its difficulty but whether the result is checkable in-process — substitute the root back,
differentiate the antiderivative back, sample the simplification numerically. That is what
work/casbenchandwork/rootcheckalready do, and it is why they can be re-run on every commit
while a leaderboard cannot.So every candidate source gets two questions, not one:
- Is an item mechanically expressible as
Solve/Integrate/Limit/Simplify/Differentiate? - Can its answer be verified without trusting a human-written key?
A source that fails (1) is a mining source at best: we read it to learn which shapes students
meet, then build our own instances. A source that passes both is a corpus we can run.
2. Tier A — CAS-native test suites. Run these.
These are aimed at computer algebra systems, not at language models, and they are the highest-value
material by a wide margin.Rubi's integration test-suite. Over 72,000 integration problems, organised by the form of the
integrand, each stored with the integration variable and an optimal antiderivative valid for all
real and complex values — and published in Axiom, Maple, Mathematica and Maxima syntax as zip files:
https://rulebasedintegration.org/testProblems.html. The top-level split (algebraic functions,
exponentials, logarithms, trig, inverse trig, hyperbolic, …) is a ready-made taxonomy for
int:categories incoverage.md, which currently has 13 hand-written ones. The site calls the
suite "freely available" but states no licence, so: usable out-of-tree inwork/today,
not vendored into an MIT repo until someone gets that in writing.Nasser Abbasi's Computer Algebra Independent Integration Tests,
https://www.12000.org/my_notes/CAS_integration_tests/index.htm — last compiled 2025-08-31, running
the suite above against Maple, Mathematica, Rubi, Fricas, Sympy, Maxima, XCas and Giac at the same
time. This gives us something no self-made corpus can: a per-problem external signal of whether an
integral is reasonably solvable at all. An integral that six systems get and we miss is a defect
with a known ceiling; one that nobody gets is not our bug. There is a sibling suite for ODEs
(https://www.12000.org/my_notes/CAS_ode_tests/index.htm) which becomes relevant the day we have an
ODE solver.Wester's CAS capability review (1999), the paper the whole practice descends from:
http://www.math.unm.edu/~wester/cas/book/Wester.pdf. SymPy encodes it as
test_wester.py
with a documentation page at https://docs.sympy.org/latest/modules/polys/wester.html. Its value here
is breadth over difficulty: it enumerates areas a CAS is expected to have at all, so it doubles as
the gap list for #717 and as evidence for #746's argument about what a platform owes its callers.
(I did not verify the often-quoted problem count, so I am not quoting it.)Licensing trap, stated once. AngouriMath is MIT. Maxima's test suite is GPL and SymPy's is
BSD-3; neither's test data can be lifted into this repo without dragging its terms along. Reading
them to learn what to test is fine. Copying them in is not.
3. Tier B — permissively licensed problem sets. Mine these.
Source Size Licence (as published) Use to us MATH (Hendrycks) 12,500 competition problems MIT Its algebra / intermediate-algebra / precalculus subsets are the closest large pool to our shape Omni-MATH 4,428 olympiad problems, 33+ sub-domains, 10 difficulty levels Apache-2.0 Upper bound of "competition"; mostly proof-shaped, mine sparingly NuminaMath ~860k problems with worked solutions Apache-2.0 Largest mining pool; the solutions name the manipulation steps, which is what we want U-MATH 1,100 university-level problems, six subjects, ~20% multimodal MIT The best match for the "tertiary education" half of this issue DeepMind mathematics_dataset56 generator modules Apache-2.0 See below — this is the important one mathematics_datasetis not a corpus, it is a generator: modules forcalculus__differentiate,
polynomials__expand,polynomials__simplify_power,algebra__linear_1dand 50-odd more, each able
to emit unbounded instances with the answer known by construction. That is exactly the shape that has
worked in this repo already —work/README.mdrecords thatrootcheckcan see missing roots
because it multiplies polynomials up from known factors, and that this found #744 and the
incompleteness behind #272, which substitution-checking structurally cannot see. A generated corpus
scales, cannot be contaminated, and carries no copyright. This should be the spine of the goal, and
the fixed corpora should be the ribs.
4. Tier C — the AI benchmarks proper. Mostly not for us.
HLE — 2,500 questions, MIT-licensed, text and image, published at
https://huggingface.co/datasets/cais/hle. Mathematics is its largest slice, but the items are
research-level short-answer and multiple-choice questions whose difficulty lives in knowing which
mathematics applies, not in carrying out a manipulation. There is close to nothing here a CAS can be
scored on. It also ships a canary string and asks explicitly never to be included in training
corpora — worth respecting even for a test corpus, so: reference it, don't vendor it. Verdict:
interesting, not a target.FrontierMath (Epoch AI) — 300 problems in tiers 1–3 plus 50 at tier 4, deliberately held out and
not public: https://epoch.ai/frontiermath/tiers-1-4/about. Unusable directly, but instructive: Epoch's
own 2026 audit corrected errors in a large fraction of the set (their write-up puts it at 42% of
problems, https://epoch.ai/benchmarks/frontiermath-tiers-1-3-v2). A hand-built key is a liability at
scale. Every point in our corpus should be machine-verifiable, and where it cannot be, it should be
labelled as trusted-by-hand.MathArena — https://matharena.ai/, evaluating recurring competitions (AIME, HMMT, USAMO, BRUMO,
SMT) immediately after each one runs, precisely to stay uncontaminated. For us its value is the
calendar: it is a maintained feed of which competitions have just published, which is the
monitoring input this issue asks for and which nobody should be re-discovering by hand.Formal-proof benchmarks — PutnamBench (1,709 Putnam problems formalised in Lean 4 / Isabelle,
some in Coq), miniF2F (244 + 244), ProofNet (371 undergraduate analysis/algebra/topology problems).
Not scoreable by us today. Worth knowing they exist, because a machine-readable statement of a
theorem is the format #746's derivation objects would eventually want to emit against.
5. Tier D — curriculum sources. These give the taxonomy, not the items.
This is where "primary education all the way to tertiary" actually gets pinned down, and where the
copyright line matters most.- OpenStax (Prealgebra through Calculus vol. 1–3) — openly licensed, but check per book: editions
differ between CC BY 4.0 and CC BY-NC-SA 4.0, and OpenStax states the books may not be ingested
into LLMs without permission. Use for structure and topic ordering. - Exam-board specifications and released papers — the AP Calculus AB/BC Course and Exam
Description plus the three most recent years of released free-response questions are published free
by the College Board (https://apcentral.collegeboard.org/courses/ap-calculus-bc/exam); Cambridge,
Edexcel, AQA, IB and the Dutchexamenblad.nlsyllabi are likewise public documents. The papers
are copyrighted. - Competition archives — IMO problems and shortlists at https://www.imo-official.org/problems.aspx
(copyright IMO). The AoPS wiki has not settled on a documentation licence
(https://artofproblemsolving.com/wiki/index.php/AoPS_Wiki:Copyright) and its own policy warns
contributors off importing text — so it is a reading source only.
The line, concretely. "A B-level paper expects you to solve
|ax + b| < cforx, and to
integratex·e^(kx)by parts" is a fact about a curriculum and we can write it down. The forty
worked exercises that teach it are an authored selection and we cannot copy them. So the workflow
below mines shapes and generates instances. That is not a compromise forced by licensing; it is
the approach that has already outperformed corpora in this repo.
6. Proposed workflow
Six stages, each leaving a re-runnable artifact rather than a claim.
1.
work/syllabus.md— the grid. One row per cell of level × topic × shape, from primary
arithmetic to tertiary analysis. A cell is(level, topic, shape template, expected answer kind, source that says it belongs at this level). Built from Tier D specs plus the Rubi and Wester
category trees, which between them cover far more of the integral and algebra space than our current
41coverage.mdcategories. The grid is the definition of "full coverage" — without it, "full" is
unfalsifiable.2.
work/examgen— generators per cell. Same pattern asrootcheckandsimpsweep: build the
instance from a construction whose answer is known before the library is called, so the harness can
detect a missing or incomplete answer and not merely a wrong one. Where an answer cannot be known
by construction, fall back to numeric verification: sample at random points including negative ones,
relative tolerance, and treatNaNas a failure only where the original had a value.3. Verification ladder, strongest first. Known-by-construction → invertible check (substitute
roots back, differentiate antiderivatives back) → numeric sampling → external agreement (12000.org's
published per-problem results,work/libcomparefor Math.NET Symbolics and Symbolism, SymPy for
#717) → hand-checked, explicitly labelled as such and kept small.4. Nuance capture — the part this issue asks for by name. Each cell records what the school-level
answer hides: domain restrictions, principal branches,±, real-vs-complex root counts, division by a
possibly-zero factor. We already emit this:coverage.mdhas
(x^2 - 1)/(x - 1) → x + 1 provided not x - 1 = 0. Turning those into a checked column means this
goal simultaneously produces the test set for #746's claim that answers should carry their conditions.5. Reporting — extend
coverage.mdwith a level axis. Alevel × topictable with the
solved / unevaluated / wrong split kept separate, because they are different kinds of news: a
wrong answer stops the line, an unevaluated one is a feature request. Only then does "we can do a
tertiary calculus paper" become a number someone can re-derive.6. Monitoring — a scheduled re-run, not a resolution. The issue asks for tracking, so the loop
needs a trigger list: MathArena's competition feed, the AIME/HMMT/Putnam/IMO calendar, new exam-board
specification versions, new Rubi releases and each summer's 12000.org rebuild. Quarterly re-run;
every new miss enters triage like any other defect.Rules to keep it honest, in the spirit of
AGENTS.md:- No claim that a paper is passable from spot checks. The number comes from the harness or it is not
made. unevaluatedandNaNstay distinct; conflating them is a wrong answer, not a rounding of one.- Nothing copyrighted is vendored. Every corpus row carries a source and a licence column, and
anything non-permissive is referenced by URL with a generated equivalent standing in for it. - Growth of the grid is allowed to reduce the coverage percentage. A goal issue that can only
improve its own metric is measuring the metric, not the library.
7. How this sits against the other goals
- Goal: Math OS — a ten-year vision for AngouriMath as an open mathematical reasoning platform #746 — this is that vision's evidence base. Worth noting that at Rubi scale the corpus also
makes the "split common-use from edge cases" suggestion in that thread measurable: we would be
able to say which rules only ever fire for a fraction of a percent of real problems, instead of
guessing. The same corpus, timed, is the speed-and-memory regression net asked for there. - Goal: feature parity with sympy #717 — Wester's review and the 12000.org tables are already cross-CAS, so the same run that
scores us produces the SymPy parity column for free. - we need more integral solvers #233 —
coverage.md's remaining misses are all integrals, and two of the four are
non-elementary. Tier A is exactly the material that says how far the integrator can go.
First concrete step
I would build stage 1 and 2 for a single vertical slice — secondary-level integration, where Rubi's
category tree gives the shapes and the 12000.org results give an external ceiling — and only widen
the grid once that slice reports a number that survives a re-run.- Is an item mechanically expressible as
First slice measured: textbook integration, 536/1774
Stage 1 and 2 of the workflow above, built for one vertical slice. The result is worth
posting on its own, because it disagrees sharply with the number this project currently
quotes itself.What was run
Rubi's integration test-suite has a directory called Independent test suites: problems
lifted from textbooks and papers rather than generated from a rule taxonomy — Stewart's
Calculus, Apostol, Timofeev's integration textbook, Hearn, Moses, Charlwood, Welz and
others. That is the education-level slice this issue is about, and unlike a corpus we
write ourselves, we did not choose it.Every problem carries an antiderivative, so the harness can tell a gap from an
impossible integral without anyone judging by hand. Answers are still graded our own
way — differentiate ours, compare to the integrand numerically at real sample points.
Rubi's answer decides only whether the problem is fair, never whether we are right.- 1,892 problems read
- 2 excluded: the integrand uses a function AngouriMath has no node for
- 116 excluded: the recorded antiderivative is non-elementary, so the problem is unfair
- 1,774 run, 5s budget each
Result
536 answered — 30.2%. 1,194 unevaluated, 7 wrong, 8 unverifiable, 29 timeouts.
Source Run Answered Apostol 159 52% Moses 107 50% Stewart 376 45% Hearn 259 40% Hebisch 4 25% Timofeev 666 19% Bronstein 10 10% Welz 104 3% Charlwood 50 2% Bondarenko / Jeffrey / Wester 39 0% By Rubi's own step count — an outside difficulty measure, so it says where our ceiling is
rather than where we put it:Rubi steps Run Answered 1 (table lookup) 268 57% 2–3 851 36% 4–6 432 16% 7–12 127 6% 13+ 48 6% The ceiling is not somewhere exotic. It is already visible at two rule applications.
Why this matters more than the number itself
work/casbenchreports 113/117 — 96.6% — on a corpus of 117 problems we wrote. Both
figures are correct measurements of different things, and only one of them is a
measurement of the library. A first-year calculus textbook is at 45%.This is the argument for the taxonomy-and-generator loop rather than more hand-written
cases: a corpus we choose cannot surprise us, and the first corpus we did not choose
moved the number by two thirds.What it found
Seven wrong answers, and all seven are one defect: a symbolic coefficient on
x^2makes
the quadratic integration rules answerNaN— filed as
#771. A second, cosmetic one came out
of the same run and is
#772.The other 1,194 misses are unevaluated rather than wrong, which is the honest failure
mode, and every one of them has an elementary antiderivative somebody else already found.
That is the real size of #233.Three things the harness got wrong first
Recording these because each one invented defects before it was fixed, and any
future stage of this workflow will meet them again:d/dx ln|u|carries aprovidedcondition, and aprovidednode has no numeric
value — so every answer containingabs()failed verification regardless of whether
it was right.- A real sample point can lie outside the integrand's real domain.
1/sqrt(x^2-1)
atx = 0.37is purely imaginary, andabs()in a correct antiderivative is not
holomorphic there, so the comparison is meaningless rather than failing. The
comparison is three-valued now — match, differs, unverifiable — and the unverifiable
cases are listed individually rather than counted, because a silent bucket reads as a
clean one. - Some entries list two acceptable antiderivatives, so they have five fields rather
than four; 91 were being dropped as unparseable.
The first two are the reason a self-verifying harness has to be verified itself. Four
"wrong answers" in the first run were artifacts of (1) and (2), not defects, and I had
written them up as bugs before checking.On not vendoring the corpus
The suite states no licence, so it is downloaded rather than committed, and the harness
takes a path to it. Same rule as in the survey above: mine the shapes, keep the
provenance, do not copy the problems in.Measured status for this goal, on
masterata45a7256, since it has never carried a number.On problems taken from an outside source, integration answers 30%.
work/intbenchruns Albert Rich's Rubi test-suite — 1892 problems, of which 1774 are fair (116 excluded because Rubi's own antiderivative is non-elementary, 2 because the integrand uses a node AngouriMath lacks). Answers are graded by differentiating them back and comparing numerically to the integrand, never against Rubi's answer:Answered 536/1774 (30.2%) — 1194 unevaluated, 7 wrong (6 of them
NaN), 8 unverifiable on the reals, 0 error, 29 timeout.On the corpus we wrote ourselves, it answers 97.5% —
casbenchreports 116/119 with 0 wrong, 0 error, 0 timeout.Both numbers are correct measurements of different things, and the gap between them is the most useful fact for this goal: a corpus you assemble yourself measures the corpus, not the library. Competition and past-paper problems are outside-source problems, so 30% is the figure to plan against, not 97.5%.
Three things follow that seem worth recording here rather than discovering later:
The 7 wrong answers matter more than the 1194 unevaluated ones. An unevaluated integral is a refusal a caller can see; a wrong antiderivative is not. Six of the seven are
NaN, which this library treats as "does not exist" — and I filed two separate defects this week whereNaNwas being used to mean "I could not do this" (#958, and the same confusion in the solver at #964). Those seven are worth reading individually before any of the 1194.29 timeouts at a 5-second budget are a different failure again — the answer may exist and simply cost too much, which is #896's territory.
The remaining unticked boxes on #233 are the same two problems the corpus leaves unsolved, so that issue and this goal agree about where the frontier is.
- added 11 commits that reference this issue
on Sep 9, 2026 281 remaining items
- added 15 commits that reference this issue
on Oct 9, 2026
Edit this issue to research what kinds of mathematical expressions a math student from primary education all the way to tertiary education may encounter, whether in textbooks, past papers or competitions. Proceed to extract examples and make sure we can solve them (albeit with nuances that might not be mentioned at lower education levels, that need to be mentioned specifically). New competitions, textbooks, past papers come out all the time, monitor and track them.