Skip to content

🐛 fix(incomplete_beta): continued fraction for large b and large a - #65

Open
nstarman wants to merge 2 commits into
JAXtronomy:mainfrom
nstarman:fix-incomplete-beta-large-b
Open

nstarman wants to merge 2 commits into
JAXtronomy:mainfrom
nstarman:fix-incomplete-beta-large-b

Conversation

@nstarman

@nstarman nstarman commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #64. Fixes #66. Fixes #67. Fixes #68. Fixes #73.

incomplete_beta's two series both expand (1-t)^(b-1) binomially. The coefficients (1-b)_k / k! alternate in sign and grow like b^k / k!, so at z = 1/2 the sum cancels catastrophically: 1e-11 relative error at b = 30, 1e7 at b = 100, and -1.1e44 (answer 1.1e-5) at b = 300.

Fix

Above b = 10, use the DLMF 8.17.22 continued fraction (modified Lentz, 64 fixed double steps):

  • directly below x₀ = (a+1)/(a+b+2);
  • above it, through the reflection B(a,b,z) = B(a,b) − B(b,a,1−z).

The complete B(a,b) is the sum of both fractions at x₀. Both terms are positive, so nothing cancels. It is deliberately not exp(jax.scipy.special.betaln), which is only good to 3e-7 at (a, b) = (8, 10).

Accuracy (against mpmath, a ∈ [0.1, 50])

b worst relative error
10–1e3 1e-13
1e5 4e-12
1e7 5e-10

The last two rows are limited by rounding, not truncation, and grow like b·ε. Past b ≈ 1/ε the switch point rounds to zero, and the result is deliberately nan rather than a confident wrong answer. The tests assert rtol 1e-12 up to b = 100 and 1e-11 up to 1e4, against mpmath. SciPy isn't used as the reference, because beta * betainc is itself only good to about 1e-11 at b = 1e4.

Cost, measured against main (1e5 points, x64, CPU)

main this PR
value, b ≤ 10 2.6 ms 2.5–3.5 ms (noise)
value, b > 10 2.9 ms (wrong above b ≈ 50) 17 ms
vmap over scalar z, b > 10 2.5 ms (wrong) 17 ms
a/b gradient, b = 1.5, traced 40 ms 46 ms
compile ∂²/∂b², traced b 1.5 s 3.4 s

Concrete a and b pick their branch in the public wrapper, before jax.custom_jvp traces them, so even under jit only that branch is compiled. The extra compile time and the 15% gradient overhead only apply to a traced a or b, which goes through the cond.

A traced b goes through lax.cond, which runs one branch. Under vmap that becomes a select that runs both, so each branch is handed a b it is safe for, and gradients stay finite (tested). The scan counter is an integer arange, so float32 input stays float32.

Tests and docs

  • Large-b accuracy against mpmath, continuity across b = 10, a/b derivatives in both modes (against mp.diff, accurate to 1e-15), and finite gradients under vmap across the switch.
  • A new test_incomplete_beta_large_b benchmark.
  • accuracy-and-domains.md, why-not-upstream.md, the registry notes and the regenerated coverage table are updated.
  • 335 tests pass locally: beta, registry, smoke, and the module and doc doctests.

Also: #68 (mixed partial at small z)

With b traced, the z-rule's factor (1-z)^(b-1) is now exp((b-1)·log1p(-z)). Before, its b-derivative went through log of a rounded 1 - z, which kept only about 1e-16/z relative precision: 5e-9 at z = 1e-8 and 2e-5 at 1e-12. A concrete b keeps pow, which XLA simplifies for constant exponents. The regression test covers both the series and the continued-fraction regimes, and b = 1 at z = 1 still gives exactly 1.

Also: large a

_large_z expands (1-u)^(a-1), so it has the same cancellation in a. Measured worst over b ∈ [-2.5, 10]: 6e-14 at a = 8, 4e-11 at 16, 2e-8 at 24, and wrong by about 50.

Above a = 8 with b ≥ 0.1, the continued fraction now takes over: 5e-13 or better up to a = 100, tested to rtol 1e-12 up to a = 50.

Large a with b < 0.1, including every b ≤ 0: the reflection doesn't apply here, because B(a,b) diverges. A third method, _large_a, handles it:

  • below s = a/(a+2), the continued fraction directly (below its switch point for b < 1/a);
  • above s, the binomial expansion the series already uses, but over u ∈ [1−z, 2/(a+2)] instead of [1−z, 1/2], so its terms fall like 2^m/m! instead of growing like (a/2)^m/m!.

Against mpmath over a ∈ [8.5, 100], b ∈ [−50, 0.1) and z up to 1 − 1e−12, the worst error is 8e-14. Dispatch is now a three-way lax.switch, still static for concrete parameters.

That expansion is factored out of _large_z, which also fixes two bugs it had on main:

  • Near-pole truncation. The near-pole expansion had 4 terms: 1.2e-12 off at b = −1e−4, z = 1 − 1e−15, and worse for larger |log(1−z)|. It now has 8 terms, evaluated once after the scan for the single step that can be near a pole, so the series is no slower.
  • nan at z = 1 for 0 < b < 1e−3. B(2, 1e−4, 1) was nan; it's now 9999.0001.

Also: two JVP fixes

  • Parameter-gradient benchmark (CodSpeed: 30.7 → 67.2 ms). The JVP rule evaluated the primal on its own and again inside the a/b jvp. With straight-line code XLA merges the two copies; with one lax.cond per copy it can't, so everything ran twice. The primal now comes from the same jvp: 2.34 ms locally, against main's 2.54 ms.
  • nan derivatives at z = 1. With b traced, the incomplete_beta: d/db d/dz loses ~1e-16/z relative precision as z -> 0 #68 form exp((b-1)·log1p(-z)) is -inf inside at z = 1, and clamping it differentiated as 0 · -inf, so the second and third z-derivatives were nan. A double where now hands z = 1 to pow, which matches the concrete-b path at every order (tested under jit).

Also: #66 and #67

Audit fixes

An independent audit of this PR found the following, all fixed with tests:

  • Integer z with b > 10 returned 0. A dtype cast truncated x₀ to zero.
  • Precision loss in the reflected term. It took log of a rounded 1 − z, scaled by b. The front factor z^a (1−z)^b is the same under the swap, so it's now computed once from z.
  • Premature underflow. The front factor flushed to zero before the fraction multiplied it back: B(301, 1000, 1) returned 0 instead of 3.9e-307. Terms are now combined in log space.
  • Too few steps. 32 steps were short just above x₀ at small a; it's now 64.
  • vmap over scalar z was about 15x slower than main. It computed the complete beta for every point; that's now done once per call.
  • "A concrete b compiles one branch" was only true eagerly. The branch is now chosen before the custom_jvp, and passed to the core as a static argument.
  • Reverse-mode overhead through the cond. Checkpointing both branches fixed it.
  • bfloat16 was 6% off. Half precision is now evaluated in float32.
  • Weak tests. Each guard and fix is now caught by some test when reverted (mutation-checked).

Second audit

A second independent audit found the following, now fixed with tests:

  • Dtypes (incomplete_beta: TypeError for float32 z with a float64 b (scan carry dtype) #66, completed). a, b and z are promoted together. That fixes a crash new in this PR (float64 a with float32 b), integer a with float32 z, float32 a with float64 b, and gradients when the other parameter is an integer.
  • Near-pole cancellation. The one step with |b+m| < 1/2 is now evaluated as w^s·Δ·expm1(sΔ)/(sΔ). Values and every b-derivative are about 1e-15, in either dtype. Before, the error was up to 5.8e-12 in value, 1.5e-3 in the third derivative, and 3.5e-4 in float32.
  • Overflow. No intermediate overflows while the result is representable, a true overflow returns +inf (never nan), and a concrete b = −1100 no longer raises OverflowError.
  • Mixed derivative at z = 1/2. ∂z∂b was halved there; it's now symmetric.
  • Half precision. a and b are also computed in float32 now; bfloat16 was 6.5% off.
  • Dead code. The large-a | pole guard is removed.

One more gap, which the audit didn't report but which surfaced while testing: on main, the series is wrong for very negative b: 6e-12 at b = −10, 8% at −50, garbage by −200. For a ≤ 8 and b < −5, the continued fraction is now used directly (128 steps; no reflection needed). Worst error is 4.4e-13 over b ∈ [−1100, −5.5].

CI. The Python 3.11 job has been killed for running out of memory on every commit since the large-a work. JAX keeps every compiled program for the session, and one test process peaked at 5.9 GB on a 16 GB runner shared by four workers. tests/unit/conftest.py now clears JAX's caches after a test once that worker passes 2 GB. That brings the peak to 2.6 GB for about 20% more runtime. Emulated locally on 3.11 with coverage and four workers: 2401 passed in 7m23s.

Known cost, documented rather than fixed: vmap over a or b runs every branch on every lane. 64 parameter pairs take 35 ms, against 2.5 ms on main. A Python loop of static calls is faster here, about 15 ms. Fixing this properly needs a custom batching rule, which is left as a follow-up.

Overlap with #63

#63 (closed form for a == 1) touches the same file. Whichever lands second needs a small conflict resolution in beta.py and the accuracy page; the two changes are otherwise independent.

🤖 Generated with Claude Code

@nstarman
nstarman requested review from ColmTalbot, adrn and cgiovanetti and a balanced review from Copilot October 9, 2026 00:01
@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.16908% with 10 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/spexial/_src/beta.py 95.16% 5 Missing and 5 partials ⚠️
Files with missing lines Coverage Δ
src/spexial/_src/registry.py 100.00% <ø> (ø)
src/spexial/_src/beta.py 95.65% <95.16%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It adds substantial new numerical code (continued fraction, reflection, and NaN-safe cond/vmap dispatch with autodiff implications) whose correctness warrants final human verification.

2 open findings
What changed in this PR

This PR fixes catastrophic cancellation in incomplete_beta for large positive b (GH-64). The two existing Taylor series both expand (1-t)^{b-1} binomially, whose coefficients alternate and grow like b^k/k!, so above b ≈ 50 the sum loses all accuracy (e.g. -1.1e44 for 1.1e-5 at b=300). Above b = 10 the implementation now switches to the DLMF 8.17.22 continued fraction (modified Lentz, 32 fixed double steps), evaluated directly below x₀ = (a+1)/(a+b+2) and via the reflection B(a,b,z) = B(a,b) − B(b,a,1−z) above it, with the complete B(a,b) formed as the sum of two positive continued-fraction terms (no cancellation). The implementation verified cleanly against Numerical Recipes' betacf and against mpmath.

Changes:

  • New continued-fraction path (_continued_fraction, _cf_direct, _large_b) used for b > 10; the two series are refactored into _series, and _incomplete_beta_impl dispatches concrete-vs-traced b (concrete picks one branch; traced uses lax.cond with a jnp.where clamp so vmap's select keeps both branches finite for autodiff).
  • Tests: large-b accuracy vs mpmath, continuity across the b = 10 switch, a/b derivatives in both AD modes, finite vmap gradients across the switch, plus a new benchmark.
  • Docs/registry notes, accuracy-and-domains, coverage table, and why-not-upstream updated to record the continued fraction.
File Description
src/​spexial/​_src/​beta.py Adds the continued-fraction path, reflection, and concrete/traced branch dispatch; refactors the two series into _series.
tests/​unit/​test_beta.py Adds large-b accuracy, continuity, parameter-derivative, and vmap finite-gradient tests.
benchmarks/​test_beta.py Adds a test_incomplete_beta_large_b benchmark.
src/​spexial/​_src/​registry.py Updates the incomplete_beta coverage note to mention the continued fraction.
docs/​reference/​accuracy-and-domains.md Documents the large-b continued fraction, domain, and tolerances.
docs/​reference/​coverage.md Regenerated note mirroring the registry change.
docs/​explanation/​why-not-upstream.md Mentions the b > 10 continued fraction alongside the series.

🧠 Review effort: Balanced


💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/unit/test_beta.py Resolved
Comment thread src/spexial/_src/beta.py Resolved
@codspeed

codspeed Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will degrade performance by 2.86%

1 warning, 1 note

⚠️ 67 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

ℹ️ 82 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 3 improved benchmarks
❌ 4 regressed benchmarks
✅ 73 untouched benchmarks
🆕 2 new benchmarks
⏩ 16 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ test_grad_incomplete_beta_autodiff 10.2 ms 13.3 ms -23.49%
❌ test_polylog_scalar[large-z2] 866.8 µs 1,075 µs -19.37%
❌ test_incomplete_beta_non_positive_b[-1.0] 1.5 ms 1.8 ms -13.44%
❌ test_incomplete_beta_non_positive_b[0.0] 1.5 ms 1.7 ms -12.9%
⚡ test_polylog_scalar[small-z0] 973.2 µs 724.2 µs +34.37%
⚡ test_polylog_scalar[intermediate-z1] 1,060.8 µs 921.7 µs +15.09%
⚡ test_zeta_vector[positive-n0] 710.5 µs 626.2 µs +13.45%
🆕 test_incomplete_beta_large_b[1000.0] N/A 4.3 ms N/A
🆕 test_incomplete_beta_large_b[50.0] N/A 4.2 ms N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing nstarman:fix-incomplete-beta-large-b (dd2b96b) with main (a65ee5a)

Open in CodSpeed

Footnotes

  1. 16 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@nstarman nstarman changed the title 🐛 fix(incomplete_beta): continued fraction for b > 10 🐛 fix(incomplete_beta): continued fraction for large b and large a Oct 9, 2026
@nstarman nstarman added this to the v0.3.0 milestone Oct 9, 2026
@nstarman
nstarman requested a balanced review from Copilot October 9, 2026 01:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes numerical correctness of a core special function across wide domains with subtle continued-fraction, reflection, and vmap-gradient-safety logic that warrants human numerical verification.

0 open findings

2 resolved since last review
Previously missed (1)

In code that hasn't changed since last review

Low severity Test name claims regularized form but uses unregularized mpmath

tests/​unit/​test_beta.py:242

This test's name says "matches_the_regularized_form", but it does not use the regularized form as its reference — it compares against mp.betainc(a, b, 0, zi), which is mpmath's unregularized incomplete beta (the docstring even stresses "The reference is mpmath, not beta * betainc"). In this same file, test_matches_the_regularized_form (line 52) genuinely uses scipy_beta(a, b) * scipy_betainc(a, b, z), so "the regularized form" already has a specific, different meaning here. The parallel large-a test added in this PR is named test_large_a_matches_mpmath (line 340). Consider renaming this one to match that convention so the name reflects the actual reference.

🧠 Review effort: Balanced

nstarman added a commit to nstarman/spexial that referenced this pull request Oct 16, 2026
…ional-only

Every private function in beta.py, and the nested `scan` steps, now ends
its parameters with `/`, matching JAXtronomy#65.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nstarman

Copy link
Copy Markdown
Collaborator Author

Ping @cgiovanetti :)

@cgiovanetti

Copy link
Copy Markdown
Collaborator

Sorry @nstarman ! I'm super swamped right now and won't have the bandwidth for this for another couple of weeks. I've set a reminder for myself for the week of November 2nd to take a look if you don't mind the wait

@nstarman

nstarman commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up: custom batching rule for vmap over a/b

I looked into implementing the custom batching rule mentioned above ("left as a follow-up") and want to record why it's a bigger lift than it looks, rather than just leaving it unimplemented silently.

The setup. _dispatch's traced-a/b path picks its branch at runtime via lax.switch, with the branch functions (_incomplete_beta_core) each carrying their own custom_jvp. _dispatch itself has no custom differentiation rule — it relies entirely on cond_p's native jvp/transpose. That's deliberate: putting a custom JVP around the switch is exactly what, in an earlier revision, left an untransposable stop_gradient in the linearized program once batched, breaking grad(vmap(...)).

Why custom_vmap can't just be bolted on. jax.custom_batching.custom_vmap requires pairing with custom_jvp or custom_vjp — unpaired, it breaks any reverse-mode AD through the function, vmap or not (confirmed empirically). I tried both pairings against the real code, testing every combination (grad(vmap), vmap(grad), direct jax.jacfwd, jax.jacfwd(jax.jacrev(...)) for Hessians, plain grad, jit(grad)):

  • custom_vjp pairing breaks jax.jacfwd(sp.incomplete_beta, ...) outright (TypeError: can't apply forward-mode autodiff (jvp) to a custom_vjp function) — and that's a currently-passing test (test_large_b_parameter_derivatives calls jacfwd directly).
  • custom_jvp pairing, with the new JVP rule computed via jax.jvp(_dispatch, ...) (reusing the existing switch), reproduces the exact historical stop_gradient/transpose bug under grad(vmap(...)) — batching a "linearize-a-switch" computation is exactly the pattern the original fix avoided.
  • Swapping the JVP rule to a switch-free, jnp.where-based reconstruction fixes correctness everywhere — but then every traced-but-non-batched a/b gradient (the common, already-benchmarked case — test_grad_incomplete_beta_parameters) runs all 4 branches instead of 1, a real regression to a documented optimization ("a traced a or b goes through lax.switch, which compiles all four and runs one").
  • The only way to get both (fast switch when not batched, safe select when batched) is to detect "am I under vmap right now" from inside the JVP rule. An ndim/shape check doesn't work — vmap hides the batch axis from code running inside the vmapped function. The only signal that works is jax.extend.core.unsafe_am_i_under_a_vmap_DO_NOT_USE(), which is explicitly named and documented as unstable/discouraged by JAX itself. Its failure mode if it ever changes is silently wrong gradients, not an error — a bad trade for a library this focused on numerical correctness.

Conclusion: a correctness-preserving custom batching rule, using only stable public JAX API, isn't achievable here without either regressing an existing optimized/benchmarked path or depending on something JAX's own maintainers flag as unsafe. Leaving this documented as a known cost (as the PR already does) rather than attempting it seems like the right call unless someone finds a cleaner primitive-level approach.

Rebase onto upstream/main, which merged JAXtronomy#63 (the `a == 1` closed form)
after this branch was cut. Both PRs restructured the same region of
beta.py independently -- JAXtronomy#63 made `incomplete_beta` a plain wrapper
around `_incomplete_beta_core`, this one added branch dispatch
(`_branch`/`_dispatch`/`lax.switch` over `_SERIES`, `_CONTINUED_FRACTION`,
`_LARGE_A`, `_NEGATIVE_B`) -- so this squashes the 17 commits that grew
on the old base into one, reconciled against JAXtronomy#63's structure:

- `incomplete_beta` checks the `a == 1` closed form first (cheapest,
  exact for any `b`), then falls through to branch dispatch for
  everything else; a traced `a` cannot be inspected and falls through
  too, even at 1.
- `_EXPRL_COEFFS`, previously defined separately by each PR for the same
  `expm1(x)/x` series, is now shared between `_expm1_over` and `_a_eq_1`.
- `_incomplete_beta_core` and its benchmarks/registry probes take the
  `a == 1` closed form's structure (a 3-arg call before JAXtronomy#63) into account
  for the branch-aware 4-arg signature this PR needs.
- `test_nan_past_one_over_eps`'s `a = 1` case is dropped: the closed
  form has no switch point to round away past `b ~ 1/eps`, so it stays
  exact where the continued fraction degrades to `nan` by design.
- `docs/reference/accuracy-and-domains.md` documents both the `a == 1`
  closed form and the branch dispatch regimes together.

No functional change to either PR's own logic; 539 tests across
test_beta.py, test_beta_regimes.py and test_registry.py pass, plus the
module's doctests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@nstarman
nstarman force-pushed the fix-incomplete-beta-large-b branch from 4001ce1 to be0a47c Compare November 3, 2026 09:06
Copilot AI balanced review requested due to automatic review settings November 3, 2026 09:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

A critical active finding remains in src/spexial/_src/beta.py where _branch uses Python if on ScalarLike values that can be 0-d jax.Array, causing invalid array-boolean conditionals and likely runtime errors for common concrete JAX inputs.

1 open issue

Review effort: Balanced

Comment thread src/spexial/_src/beta.py Resolved
Copilot AI balanced review requested due to automatic review settings November 3, 2026 10:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🔵 Needs a closer look

The change is a large, numerically sensitive refactor of incomplete_beta (new multi-regime branching, continued-fraction implementation, and custom JVP/fixture updates) whose stability and edge-case accuracy need domain-expert sanity checks beyond the tested grids.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@nstarman

nstarman commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Ping @cgiovanetti when you get a chance. I think we can release the next version after this PR.

@nstarman

Copy link
Copy Markdown
Collaborator Author

Hi @cgiovanetti :)

@cgiovanetti

Copy link
Copy Markdown
Collaborator

Sorry @nstarman ! Just got back from a bunch of travel and I've got a couple deadlines coming up--will try to get to it within the next two weeks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants