Implementing a file based queue

Question

I have an in memory bounded queue in which multiple threads queue objects. Normally the queue should be emptied by a single reader thread that processes the items in the queue.

However, there is a possibility that the queue is filled up. In such a case I would like to persist any additional items on the disk that would be processed by another background reader thread that scans a directory for such files and processes the entries within the files. I am familiar with Active MQ but prefer a more light weight solution. It is ok if the "FIFO" is not strictly followed (since the persisted entries may be processed out of order).

Are there any open source solutions out there? I did not find any but thought I would ping this list for suggestions before I embark on the implementation myself.

Thank you!

Answer 1

You could use something like SQLLite to store the objects in.

Answer 2

Take a look at http://square.github.io/tape/, and its impressive QueueFile.

(thanks to Brian McCallister's "The Long Tail Treasure Trove" for pointing me at that).

Answer 3

EHCache can overflow to disk. It's also highly concurrent, though you dont really need that

Answer 4

Why is the queue bounded? Why not use a dynamically expandable data structure? That seems much simpler than involving the disk.

Edit: It's hard to answer your question with out more context.

Can you clarify what you mean by "run out of memory"? How big is the queue? How much memory do you have?

Are you on an embedded system with very little memory? Or do you have 2 GB or more of stuff in the queue?

If either is true, you really aught to use a "swappable" data structure like a BTree. Implementing one your self for one queue seems like overkill. I would just use an embedded database like SQL lite.

If neither of those us true, then just use a vector or a linked list.

Edit 2: You probably don't need a BTree or a database. You could just use a linked list of pages. But again, I have to ask: is this necessary?

Or, if you are willing to process things non serially, why not have multiple reader threads all the time?

Ultimately though I don't think your proposal is the way to go.

Answer 5

You could embed berkley db java edition for keeping queue elements in files.

You can look at working example here: http://sysgears.com/articles/lightweight-fast-persistent-queue-in-java-using-berkley-db

Hope this helps

Answer 6

The most performant and GC friendly solution I've found by now is Chronicle Queue. It has extremely low write latency, order of tens of nanoseconds, several grades of magnitude lower than MapDB or SQLite.

Answer 7

MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap-memory. It is a fast and easy to use embedded Java database engine.

https://github.com/jankotek/MapDB

http://www.mapdb.org/