Repository navigation
[common] Reject out-of-range bloom filter items and fpp options - #10108
Conversation
The bloom filter file index reads items and fpp from table options with no bounds. A percentage-shaped fpp such as 10 makes every write fail with a bare NegativeArraySizeException that names neither the option nor the column; fpp=0 overflows the same way; fpp=1 or a negative value silently builds a one-byte filter that prunes nothing; and a very large items overflows the bit-set size. Reject fpp outside (0, 1) and non-positive items at the writer -- the read path never consults them, so an already-written table stays readable -- and guard the bit-count overflow in BloomFilter64, so a bad option fails at write time with a message naming it.
4ab826d to
916fb0c
Compare
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
The reported failure has an end-to-end impact: malformed Bloom filter options currently fail during writes with an opaque allocation error or silently produce ineffective indexes. This change rejects invalid fpp and non-positive items at writer construction and checks the computed bit count before allocation. It leaves deserialization of existing index files untouched. I checked the numeric boundaries, including NaN/infinity behavior at the writer guard and the final byte-alignment arithmetic; I found no actionable regression.
Local verification: BloomFilterFileIndexTest (3 tests) and BloomFilter64Test (1 test) passed, including the new invalid-option cases and existing write/read behavior. git diff --check passed.
Requirement fit: SUPPORTED. Implementation: CLEAN.
2 participants
Purpose
The bloom filter file index reads
itemsandfppfrom table options with no bounds. A percentage-shapedfpplike10makes every write fail with a bareNegativeArraySizeException;fpp = 0overflows the same way;fpp = 1or negative silently builds a one-byte filter that prunes nothing; a largeitemsoverflows the bit-set size.Reject
fppoutside(0, 1)and non-positiveitemsin the writer (the read path never consults them, so existing tables stay readable), and guard the bit-count overflow inBloomFilter64. A bad option now fails at write time naming it. Mirrors #9765 for range-bitmapchunk-size.Tests
BloomFilterFileIndexTest.testRejectsInvalidOptions:fpp= 10 / 0 / 1.0 / -0.1,items= 0,items = Integer.MAX_VALUE, plus a valid config. Fails on master, passes here.Written with Claude Code; verification is mine.