File backed Java map

Question

Is there a simple way of having a file backed Map?

The contents of the map are updated regularly, with some being deleted, as well as some being added. To keep the data that is in the map safe, persistence is needed. I understand a database would be ideal, but sadly due to constraints a database can't be used.

I have tried:

Writing the whole contents of the map to file each time it gets updated. This worked, but obviously has the drawback that the whole file is rewritten each time, the contents of the map are expected to be anywhere from a couple of entries to ~2000. There are also some concurrency issues (i.e. writing out of order results in loss of data).

Using a RandomAccessFile and keeping a pointer to each file's start byte so that each entry can be looked up using seek(). Again this had a similar issue as before, changing an entry would involve updating all of the references after it.

Ideally, the solution would involve some sort of caching, so that only the most recently accessed entries are kept in memory.

Is there such a thing? Or is it available via a third party jar? Someone suggested Oracle Coherence, but I can't seem to find much on how to implement that, and it seems a bit like using a sledgehammer to crack a nut.

Answer 1

You could look into MapDB which was created with this purpose in mind.

MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap-memory. It is a fast and easy to use embedded Java database engine.

Answer 2

Yes, Oracle Coherence can do all of that, but it may be overkill if that's all you're doing.

One way to do this is to "overflow" from RAM to disk:

    BinaryStore diskstore = new BerkeleyDBBinaryStore("mydb", ...);
    SimpleSerializationMap mapDisk = SimpleSerializationMap(diskstore);
    LocalCache mapRAM = new LocalCache(100 * 1024 * 1024); // 100MB in RAM
    OverflowMap cache = new OverflowMap(mapRAM, mapDisk);

Starting in version 3.7, you can also transparently overflow from RAM journal to flash journal. While you can configure it in code (as per above), it's generally just a line or two of config and then you ask for the cache to be configured on your behalf, e.g.

    // simplest example; you'd probably use a builder pattern or a configurable cache factory
    NamedCache cache = CacheFactory.getCache("mycache");

For more information, see the doc available from http://coherence.oracle.com/

For the sake of full disclosure, I work at Oracle. The opinions and views expressed in this post are my own, and do not necessarily reflect the opinions or views of my employer.

Answer 3

jdbm2 looks promising, never used it but it seems to be a candidate to meet your requirements:

JDBM2 provides HashMap and TreeMap which are backed by disk storage. It is very easy and fast way to persist your data. JDBM2 also have minimal hardware requirements and is highly embeddable (jar have only 145 KB).

You'll find many more solutions if you look for key/value stores.