Skip to content

feat: exact, deterministic numba k-means (fast_array_utils.cluster) - #241

Open
timtreis wants to merge 15 commits into
scverse:mainfrom
timtreis:cluster-kmeans
Open

timtreis wants to merge 15 commits into
scverse:mainfrom
timtreis:cluster-kmeans

Conversation

@timtreis

@timtreis timtreis commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

This PR adds fast_array_utils.cluster with kmeans, kmeans_plusplus and KMeansResult. It gives scverse packages one shared, fast k-means implementation, instead of each package calling scikit-learn or carrying its own. The callers we found are scanpy, mantispy's PHATE landmarks, harmonypy's seeding, and several other ecosystem packages.

Contract

  • Exact. The labels are those of plain float64 Lloyd from the same initial centers. The scans rank centers with float32 or FMA dot products, and a rigorous rounding bound per center decides when a label is certain. Every center that could still be nearest is rechecked in float64.
  • Deterministic. The results don't depend on the number of threads or on the threading layer, because the work is split into fixed chunks that are combined in a fixed order.
  • Seeding is greedy k-means++ with scikit-learn's number of local trials (2 + ln k). seed_sample seeds on a random subset of rows.
  • Stopping follows scikit-learn's tol rule. max_moved also stops once a pass changes at most that many labels.
  • Inputs: dense float32 or float64 arrays, used as they are; other dtypes are converted to float64. NaN, inf and invalid k raise ValueError before any work is done. rng is keyword-only.

How it works

  • Lloyd uses Hamerly bounds. Rows that fail the bound test are scanned with FMA register tiles. For 256 ≤ k ≤ 2048 the rows are scanned per label, against only the centers that could be nearer ("ball" scans).
  • Seeding scans its candidates with FMA tiles too, and ends a candidate's scan early once no row can be closer to it.
  • Threading: each phase runs in one persistent parallel region, with chunks claimed by atomics.

Speed

Timings are medians over 5 seeds, from 16 threads on an AMD EPYC 9845, against scikit-learn's KMeans with the same thread count.

Case vs scikit-learn ours
Lloyd, 1M × 50, k=100 13.8× 39 ms
End to end, 1M × 50, k=100 3.3× 242 ms
Lloyd, 200k × 50, k=1000 12.0× 33 ms
End to end, 200k × 50, k=1000 3.4× 255 ms
Parse 300k PCs, cell types 4.7× 39 ms
PHATE landmarks, 100k × 100 float64, k=2000 1.8× 781 ms
End to end 1M, 8 threads, AVX2 3.1× 440 ms

At 1M rows the peak extra memory is 37 MB, against about 350 MB for scikit-learn. mantispy ran this branch as a drop-in for its PHATE landmark k-means (float64, d=100, k=2000; Xeon Gold 6248R, 16 threads):

  • k-means on 989k real cells (Aconcagua s1m): 85 s before, 19 s with this branch;
  • the same labels on 1, 4 and 16 threads;
  • all 38 of mantispy's PHATE tests pass.

Tests

tests/test_cluster.py covers:

  • exactness against plain float64 Lloyd, for float32 and float64 input, including adversarial data (far from the origin, high dimensions, near-duplicate centers) and a case with many centers on unclustered data, which runs the per-label scans;
  • identical results on 1 and 4 threads, with and without the transposed copy used in seeding;
  • max_moved, seed_sample, and seeds that are rows of the data;
  • NaN, inf and invalid k, for both kmeans and kmeans_plusplus;
  • two codspeed benchmarks (100k × 50, k=30 and k=1000).

The docs get a cluster page.

Open points for review

  • numba becomes a hard requirement of fast_array_utils.cluster; it could move to an extra instead.
  • The code uses llvmlite intrinsics (atomics, prefetch, vector FMA tiles, cycle counter). A review of that part for portability (ARM, Windows) would help.

timtreis and others added 2 commits November 18, 2026 06:10
Adds fast_array_utils.cluster with kmeans, kmeans_plusplus and KMeansResult: greedy k-means++ seeding and
Lloyd with Hamerly bounds, FMA register tiles and exact float64 rechecks, results independent of the thread
count. Includes tests against plain float64 Lloyd, adversarial exactness cases, determinism and input errors.
@timtreis timtreis changed the title cluster: exact, deterministic numba k-means feat: exact, deterministic numba k-means (fast_array_utils.cluster) Nov 18, 2026
@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.61%. Comparing base (5fb995d) to head (7a26d89).

Additional details and impacted files
@@             Coverage Diff             @@
##              main     #241      +/-   ##
===========================================
- Coverage   100.00%   98.61%   -1.39%     
===========================================
  Files           21       29       +8     
  Lines          592      868     +276     
===========================================
+ Hits           592      856     +264     
- Misses           0       12      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed

codspeed Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ 46 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 234 untouched benchmarks
🆕 4 new benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
🆕 test_kmeans_benchmark[128] N/A 479.8 ms N/A
🆕 test_kmeans_benchmark[8] N/A 65.4 ms N/A
🆕 test_kmeans_plusplus_benchmark[128] N/A 677.2 ms N/A
🆕 test_kmeans_plusplus_benchmark[8] N/A 53 ms N/A

Comparing timtreis:cluster-kmeans (7a26d89) with main (5fb995d)

Open in CodSpeed

timtreis and others added 2 commits November 18, 2026 07:11
…hared tile codegen and chunk-sum update

- kmeans validates once and rejects NaN/inf before seeding; kmeans_hamerly folded into kmeans; rng keyword-only in kmeans_plusplus
- documented KMeansResult and numpy-style parameters; docs/cluster.rst
- named istate slots; one _chunk_sums helper for the bounds pass and the ball scans; shared _vptr/_splat/_fma and one dot-tile body
- noqa codes for the kernels moved to per-file-ignores
- tests: one parametrized exactness test (incl. float64 high-d and a ball-scan case), no-copy seeding path, seeds are rows, NaN/inf for both functions, codspeed benchmarks; reference Lloyd without n×k×d temporaries

Results are byte-identical to the previous commit on 10 cases (incl. 1M rows, k=2000 float64); paired timing 0.970 [0.899, 0.998].
@timtreis
timtreis marked this pull request as ready for review November 18, 2026 17:08

@flying-sheep flying-sheep left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Public API looks clean. Code could be restructured, there’s a lot in a single file here.

Complexity should be managed by chopping up algorithms into clear inlined functions, e.g. a “prepare memory” - “run update step until convergence” – “construct final result” API should be identifiable at a glance and not be a huge pile of spaghetti.

I’ll take a look once you fixed the lints instead of suppressing them!

Comment thread pyproject.toml Resolved
Comment thread pyproject.toml Resolved
Comment thread pyproject.toml Outdated
Comment on lines +148 to +153
"C901",
"PLR0912",
"PLR0913",
"PLR0914",
"PLR0915",
"PLR0917",

@flying-sheep flying-sheep Nov 19, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • use @njit(inline='always') to restructure complex functions or suppress locally if that really brings no clarity
  • suppress the too-many-arguments warnings using one #noqa for each kernel that needs it

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The long kernels are split into inline="always" phases, and each kernel that needs it has one # noqa: PLR0917. A few long phases keep a local complexity noqa, because splitting them further only passes the same arrays around. According to Claude, numba 0.68 rules out the two ways to group them: keyword-only calls between kernels don't compile, and a NamedTuple of arrays loses its writes inside prange. Tried that refactor and couldn't get it not to segfault

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t get it. I know the keyword-only bug in numba, but I don’t get why passing arrays around like before 73ad392 would be a problem.

Comment thread pyproject.toml Resolved
@timtreis

Copy link
Copy Markdown
Member Author

Thanks for the feedback already! Restructured as you described, I tried fixing the lints as much as possible without complicating code paths. The API is unchanged, except that x is positional-only. Results are byte-identical to the previous head on all benchmark cases and thread counts, and speed is within noise

Comment thread src/fast_array_utils/cluster/_ball.py Outdated
reach = d1[r] + err_a[best] + err_b[best] * xs[r] # ≥ the nearest candidate’s true v
wo = l1[r] if lb1[r] != bb[r] else l2[r]
rest = (out - upper[i]) * (1 - SAFETY) # every center not scanned is at least this far from row i
best, e1, e2 = _label_ball_row(x, xnorm[i], i, r, centers, dv, srt, pe, cnorm, err_a, err_b, xs[r], d1[r], l1[r], l2[r], bb[r], lb1[r])

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is what you mean: this is as incomprehensible as the expanded version?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants