Kyoto cabinet TreeDB: memory usage grows uncontrollably until database is closed

Question

I've been trying to use kyotocabinet TreeDB (MSVC10 build) and stumbled upon a strange memory issue: basically, on each database write, memory usage grows and it doesn't drop until database is closed.

The test code looks like this:

size_t BLOCK_SIZE = 1 << 20; // use 1MB-sized records
char* test_block = new char[BLOCK_SIZE]; // allocate record data

TreeDB db;
db.open("test.db")

// add 5000 records
for (int i = 0; i < 5000; ++i)
{
    // at each call, process virtual memory usage increases by 1 MB even though i'm not allocating more memory
    // also, as expected, in x86-32 process the whole thing runs out of memory and crashes when iteration counter comes close to 2000
    db.set(&i, sizeof(i), test_block, BLOCK_SIZE);
}

db.close(); // memory usage returns to normal here
delete [] test_block;

Of course, I could open/close database when adding each record, but this introduces a huge delay (about 1 sec), which is not acceptable for my task. Also, this problem doesn't occur in HashDB, but I can't really use HashDB, as I occasionally need sequential access by key.

I've tried altering tuning parameters (tune_page_cache, tune_buckets, tune_page) but with no luck. Could please someone hint at what I'm missing here? I need to store an unpredictable amount of 100KB-10MB sized records and run it on 32-bit system.

Answer 1

Perhaps db.synchronize() would help, as it should push cached data out to the filesystem, allowing user memory to free up. If so, and you need this to happen all the time, call open with the optional argument OAUTOSYNC, although this will likely slow you down quite a lot.