Repository navigation
Federated statistics histograms count infinite samples twice #5360
Description
Activity
Opened #5383 for this. Both parts, the appended bucket and the elif on the one-bin case.
Worth adding to your writeup: when both infinities are present the negative bucket never makes it out either, since
bucketis assigned twice in that block and the positive assignment overwrites it. So the output is three bins plus one(inf, inf), with the negative infinity counted once in bucket 0 and the positive one counted twice.The order dependence you mentioned is concrete in
accumulate_hists: it keys the global histogram on the first client bins and then iterates only those when writing counts back, so a later client(inf, inf)bucket is dropped while the same bucket from the first client sticks and inflates the global total.Correction to my comment above: I closed #5383. #5361 already does this and is the better patch, and I had missed it because I read this thread and never searched the open PRs. Go with #5361.
The one observation from mine that is not in this thread: with both infinities present, the appended bucket for minus infinity never reached the output either, because
bucketis assigned twice in that block and the positive assignment overwrote it. #5361 deletes the block, so it is moot, but it explains why the symptom looked like one extra bucket rather than two.
On
mainata02743ebe1f7f42e601633685735596062fc7764,get_std_histogram_bucketsadds infinite samples to the first/last bins and then appends another degenerate infinity bin containing samples already counted.Reproduction
Actual:
4 4, despite requesting three bins for three samples.Expected: three bins and a total count of three, with negative and positive infinities included once in their respective edge buckets.
There is a related one-bin case: an
elifprevents both infinity endpoints from being added to the same bucket. Using independent endpoint checks preserves a single bucket with bounds(-inf, inf)when both are present.The duplicate counts also reach the dataframe statistics implementation and global histogram aggregation. Reproduced on macOS, Python 3.12.11 and NumPy 2.5.3, using an explicitly supplied finite histogram range. Automatic range inference with infinite values is outside this report.