Configure HAProxy for rabbitmq

Question

I want to use HAProxy as a load balancer. I want to put two rabbitmq server behind haproxy. Both the rabbitmq server are on different instance of EC2. I have configure HAProxy server by following this reference. I works but the problem is messages are not published in roundrobin pattern. Messages are publish only on one server. Is there any different configuration for my requirement?

My configureation in /etc/haproxy/haproxy.cfg

listen rabbitmq 0.0.0.0:5672
     mode    tcp
     stats   enable
     balance roundrobin
    option tcplog
    no  option clitcpka
    no option srvtcpka
     server  rabbit01 46.XX.XX.XX:5672 check
     server  rabbit02 176.XX.XX.XX:5672 check
listen  web-service *:80
          mode    http
         balance roundrobin
        option httpchk HEAD / HTTP/1.0
         option httpclose
        option forwardfor
        option httpchk OPTIONS /health_check.html
        stats enable
        stats refresh 10s
        stats hide-version
        stats scope   .
        stats uri     /lb?stats
        stats realm   LB2\ Statistics
        stats auth    admin:Adm1nn

Update:

I have made some R&D on this and found that HAProxy is round robin the connection on the rabbitmq server. for ex: if i request for 10 connections then it will round robin the 10 connection over my 2 rabbitmq servers and publish the message.

But the problem is I want to round robin the messages, not connection it should be manage by HAProxy server. i.e if i send 1000 msg at a time to HAProxy then 500 msg should go to rabbit server1 and 500 msg should go to rabbit server2. What should be the configuration that i have to follow?

Update:

I have also test with leastconn in balancing but HAProxy behavior in unexpected. I have posted that question on serverfault.com

Answer 1

Messages get published to an exchange which will route to a queue.

You probably didn't configure you queues with {"x-ha-policy","all"}. Based on the fact that the exchange routing is working on both nodes this is probably all you are missing.

Note: Pre Rabbit 3.0 you would declare a queue with the x-ha-policy argument and it would be mirrored. With rabbit 3.0 you need to apply a policy (ha-mode = all). You can set policies through the api or the api tools (rabbitmqctl, management gui). i.e.

rabbitmqctl set_policy -p '/' MirrorAllQueues '.+' '{"ha-mode": "all"}'

Answer 2

The AMQP protocol is designed to use persistent connections, meaning you won't get a new connection per AMQP message (to avoid the overhead of constantly reconnecting). This means that a load balancer such as HAProxy will not be effective in balancing out your messages - it can only help with balancing out your connections.

Thus you cannot achieve your stated goal. If, however, your actual goal is to distribute messages evenly to consumers of those RabbitMQ instances, then you can use clustering as Karsten describes or you can use federation.

Federation setup:

First you need to enable the federation plugins:

rabbitmq-plugins enable rabbitmq_federation
rabbitmq-plugins enable rabbitmq_federation_management

Then for each of your servers log on to the RabbitMQ Web UI as an administrator and go to Admin > "Federation Upstreams" > "Add a new upstream" and add the other server(s) as upstream(s).

Now you need to define a policy for each exchange/queue you want to be federated. I only managed to get federation working for queues mind you, so I would try that first. Go to Admin > "Policies" > "Add / update a policy" and add a policy that targets the queue(s) you want federated.

Answer 3

Remove the 'backup' from the server definitions.

A backup server is one that will be used when all others are down. Specifying all your servers as backup without using option allbackups will likely have untoward consequences.

Change the relevant portion of your config to the following:

listen  rebbitmq *:5672
        mode    tcp
        balance roundrobin
        stats enable
        option  forwardfor
        option  tcpka
        server  web2 46.XX.XX.XXX:5672 check inter 5000
        server web1 176.XX.XX.XX:5672 check inter 5000