Repository navigation
Conversation
Adds fast_array_utils.cluster with kmeans, kmeans_plusplus and KMeansResult: greedy k-means++ seeding and Lloyd with Hamerly bounds, FMA register tiles and exact float64 rechecks, results independent of the thread count. Includes tests against plain float64 Lloyd, adversarial exactness cases, determinism and input errors.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #241 +/- ##
===========================================
- Coverage 100.00% 98.61% -1.39%
===========================================
Files 21 29 +8
Lines 592 868 +276
===========================================
+ Hits 592 856 +264
- Misses 0 12 +12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will not alter performance1 warning
Performance Changes
Comparing |
…hared tile codegen and chunk-sum update - kmeans validates once and rejects NaN/inf before seeding; kmeans_hamerly folded into kmeans; rng keyword-only in kmeans_plusplus - documented KMeansResult and numpy-style parameters; docs/cluster.rst - named istate slots; one _chunk_sums helper for the bounds pass and the ball scans; shared _vptr/_splat/_fma and one dot-tile body - noqa codes for the kernels moved to per-file-ignores - tests: one parametrized exactness test (incl. float64 high-d and a ball-scan case), no-copy seeding path, seeds are rows, NaN/inf for both functions, codspeed benchmarks; reference Lloyd without n×k×d temporaries Results are byte-identical to the previous commit on 10 cases (incl. 1M rows, k=2000 float64); paired timing 0.970 [0.899, 0.998].
…lint ignores, mypy override for untyped numba kernels)
… the Windows clock branch from coverage
flying-sheep
left a comment
There was a problem hiding this comment.
Public API looks clean. Code could be restructured, there’s a lot in a single file here.
Complexity should be managed by chopping up algorithms into clear inlined functions, e.g. a “prepare memory” - “run update step until convergence” – “construct final result” API should be identifiable at a glance and not be a huge pile of spaghetti.
I’ll take a look once you fixed the lints instead of suppressing them!
| "C901", | ||
| "PLR0912", | ||
| "PLR0913", | ||
| "PLR0914", | ||
| "PLR0915", | ||
| "PLR0917", |
There was a problem hiding this comment.
- use
@njit(inline='always')to restructure complex functions or suppress locally if that really brings no clarity - suppress the
too-many-argumentswarnings using one#noqafor each kernel that needs it
There was a problem hiding this comment.
Done. The long kernels are split into inline="always" phases, and each kernel that needs it has one # noqa: PLR0917. A few long phases keep a local complexity noqa, because splitting them further only passes the same arrays around. According to Claude, numba 0.68 rules out the two ways to group them: keyword-only calls between kernels don't compile, and a NamedTuple of arrays loses its writes inside prange. Tried that refactor and couldn't get it not to segfault
There was a problem hiding this comment.
I don’t get it. I know the keyword-only bug in numba, but I don’t get why passing arrays around like before 73ad392 would be a problem.
…l, _tiles, _sync; public API in __init__
…are / iterate / result)
…llvmlite stubs, drop the file-wide lint ignores and the mypy override
…d passes and no timed ball scans
…easured reasons for the persistent regions
…arding layers, name slots and constants, tie test with certain ties
|
Thanks for the feedback already! Restructured as you described, I tried fixing the lints as much as possible without complicating code paths. The API is unchanged, except that x is positional-only. Results are byte-identical to the previous head on all benchmark cases and thread counts, and speed is within noise |
… slowed D3 float64)
| reach = d1[r] + err_a[best] + err_b[best] * xs[r] # ≥ the nearest candidate’s true v | ||
| wo = l1[r] if lb1[r] != bb[r] else l2[r] | ||
| rest = (out - upper[i]) * (1 - SAFETY) # every center not scanned is at least this far from row i | ||
| best, e1, e2 = _label_ball_row(x, xnorm[i], i, r, centers, dv, srt, pe, cnorm, err_a, err_b, xs[r], d1[r], l1[r], l2[r], bb[r], lb1[r]) |
There was a problem hiding this comment.
I guess this is what you mean: this is as incomprehensible as the expanded version?
This PR adds
fast_array_utils.clusterwithkmeans,kmeans_plusplusandKMeansResult. It gives scverse packages one shared, fast k-means implementation, instead of each package calling scikit-learn or carrying its own. The callers we found are scanpy, mantispy's PHATE landmarks, harmonypy's seeding, and several other ecosystem packages.Contract
2 + ln k).seed_sampleseeds on a random subset of rows.tolrule.max_movedalso stops once a pass changes at most that many labels.kraiseValueErrorbefore any work is done.rngis keyword-only.How it works
Speed
Timings are medians over 5 seeds, from 16 threads on an AMD EPYC 9845, against scikit-learn's
KMeanswith the same thread count.At 1M rows the peak extra memory is 37 MB, against about 350 MB for scikit-learn. mantispy ran this branch as a drop-in for its PHATE landmark k-means (float64, d=100, k=2000; Xeon Gold 6248R, 16 threads):
Tests
tests/test_cluster.pycovers:max_moved,seed_sample, and seeds that are rows of the data;k, for bothkmeansandkmeans_plusplus;The docs get a
clusterpage.Open points for review
fast_array_utils.cluster; it could move to an extra instead.