Persistent Blocking Queue in Java?

Question

TL;DR; I need to know if there's a lib with persistent blocking queue that performatic.

I hava a classic producer/consumers program. They share a LinkedBlockingQueue to share the data, and I use BlockingQueue#take method in the Consumers, as I need them to live forever waiting for new elements.

The problem is I have LOTS of data and I can't lose them. Even after the consumers stops, the producer can persist to generate some data. I am thinking about implementing my BlockingQueue ta uses H2 behind to store/get the data after some threshold is reached. My main problem is that I need performance and I need to consume the elements in the order they are created.

Is there an implementation of persistent blocking queue that I can use for something like this? If it doesn't, any sugestions for me to achieve something like this?

Answer 1

I would use ActiveMQ lib and Spring JMS, here is a usage example

start broker

BrokerService broker = new BrokerService();
broker.addConnector("tcp://localhost:61616");
broker.start();

read msg

ConnectionFactory cf = new ActiveMQConnectionFactory("tcp://localhost:61616");
JmsTemplate t = new JmsTemplate(cf);
Message msg = t.receive();

send message

ConnectionFactory cf = new ActiveMQConnectionFactory("tcp://localhost:61616");
JmsTemplate t = new JmsTemplate(cf);
t.send("test", new MessageCreator() {
  public Message createMessage(Session session) throws JMSException {
    return session.createTextMessage("test");
  }
});

Answer 2

You can try ActiveMQ. The ActiveMQ can write to your file system so if the producer is generating many more elements than the consumer can take you either have a lot of blocking (on whatever the upper bound of the queue is) or excessive data (if there is no upper bound to the queue).

Answer 3

Have you come across Amazon SQS its an unbounded queue and very fast, It guarantees order.How long do you want to persist the data for?

Answer 4

You can use any JMS implementation for supporting excess incoming data. This is a producer consume problem and jms is designed for this.