Kyoto TreeDB insertion slows down after 100 million records

Question

I am inserting records in Kyoto TreeDB. The key is simply a counter which I increment and the value is fixed.

After 100 million records, insertion gradually keeps slowing down. Has anyone tried to insert more records in Kyoto Cabinet?

Answer 1

You're probably getting hash collisions. It will put records into an overflow area if that happens. I believe that degenerates to a searched list.

Answer 2

HashDB is more memory efficient than TreeDB. Since you are using TreeDB you may want to tinker with the cache option to keep insertions fast.

Answer 3

Did you increase bnum and msiz accordingly?

I had very poor insert performance with a too low msiz value.