Guava Bloom Filter does not support large insertions?

Question

I was using BloomFilter in guava v.11.0.1 and it seems like I am getting an exception when my insertion is large. I tried at 10 million with 0.001 fpp, and it failed.

java.lang.IllegalArgumentException: Number of bits must be positive
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
    at com.google.common.hash.BloomFilterStrategies.checkPositiveAndMakeMultipleOf64(BloomFilterStrategies.java:72)
    at com.google.common.hash.BloomFilterStrategies.access$000(BloomFilterStrategies.java:18)
    at com.google.common.hash.BloomFilterStrategies$From128ToN.withBits(BloomFilterStrategies.java:37)
    at com.google.common.hash.BloomFilter.create(BloomFilter.java:192)
    at com.ipg.collection.BloomFilterWritable.impl(BloomFilterWritable.java:43)
    at com.ipg.collection.BloomFilterWritable.put(BloomFilterWritable.java:62)
    at com.ipg.prophet.twitter.twitflow.archive.UnzipTweetsProcessAndUpload$ProcessorConsumer.process(UnzipTweetsProcessAndUpload.java:107)
    at com.ipg.prophet.twitter.twitflow.archive.UnzipTweetsProcessAndUpload$ProcessorConsumer.run(UnzipTweetsProcessAndUpload.java:84)
    at java.lang.Thread.run(Thread.java:662)

I think at least it should support that many insertions with such a high fpp, shouldn't it?

Answer 1

Sorry about this, I'm the culprit :)

Hopefully we will be able to push the next version soon. Not the time to mention this, but there is an upside to this accident: it means we can definitely kill the current serial form of BF and its related supporting code (which was an accident itself), which I'm trying to fix for a month now - incidentally the fix to that also fixes this problem.

Edit: more information here (and in Louis' filed issue)

Answer 2

This should probably be filed as an issue on Guava, not on StackOverflow. (I confirm it, by the way; and I've mostly figured out what's going on.)

UPDATE: I've filed an issue and started a patch.