Berkeley DB File compression

Question

This is in conjunction with my previous question Click here. We are using berkeley DB for temporary storage before it is processed and stored into a relational DB.The problem arises when the size increases beyond a certain point.Now we have to either split the files into smaller ones or compress the existing files.In this question I want to ask the compression part,whether berkeley DB has any built in compression utility or do we have to do it programatically.If it is built in,then it will always be faster.

Answer 1

From here:

According to the Berkeley FAQ there are two ways of optimizing it (before compression):

1. Compact
2. Vacuum

You can also implement your own compression algorithm as shown here.

How different is the Berkeley DB VACUUM from SQLite's?

SQLite implements the VACUUM command as a database dump followed by a complete reload from that dump. It is an expensive operation, locking the entire database for the duration of the operation. It is also an all or nothing operation. Either it works, or it fails and you have to try again sometime. When SQLite finishes, the database is frequently smaller in size (file size is smaller) and the btree is better organized (shallower) than before due to in-order key insertion of the data from the dump file. SQLite, when it works and when you can afford locking everyone out of the database, does a good job of VACUUM. Berkeley DB approaches this in a completely different way. For many releases now Berkeley DB's B-Tree implementation has had the ability to compact while other oprations are in-flight. Compacting is a process wherein the B-Tree nodes are examined and, when less than optimal, they are re-organized (reverse split, etc.). The more shallow your B-Tree, the fewer lookups required to find the data at a leaf node. Berkeley DB can compact sections of the tree, or the whole tree at once. For 7x24x365 (five-nines) operation this is critical. The BDB version of compact won't adversly impact ongoing database operations whereas SQLite's approach does. But compaction doesn't address empty sections of the database (segments of the database file where deleted data once lived). Berkeley DB also supports compression of database files by moving data within the file, then truncating the file returning that space to the filesystem. As of release 5.1 of Berkeley DB, the VACUUM command will compact and compress the database file(s). This operation takes more time than the dump/load approach of SQLite because it is doing more work to allow for the database to remain operational. We believe this is the right trade-off, but if you disagree you can always dump/load the database in your code.