Programmatically add nodes to a load balancer like Haproxy?

Question

I'm very sure this problem has been solved, but I can't find any information anywhere about it...

How do sysadmins programmatically add a new node to an existing and running load balancer ? Let's say I have a load balancer running and already balancing say my API server between two EC2 instances, and suddenly there's a traffic spike and I need a third node in the load balancer but I'm asleep... It would be wonderful if I had something monitoring probably RAM usage and some key performance indicators that tell me when I should have another node, and even better if it could add a new node to the load balancer alone...

I'm confident that this is possible and even trivial to do with node-http-proxy and distribute, but I'd like to know if this is possible to do with HAproxy and/or Nginx... I know Amazon's elastic load balancing is probably my best bet but I want to do it on my own (I want to spawn instances from rackspace, EC2, Joyent and probably others as it's convenient).

Once again, spawning a node is easy, I'd like to know how to add it to haproxy.cfg or something similar with Nginx without having to reload the whole proxy, and doing that programatically. Bash scripting is my best bet for this but it still does have to reload the whole proxy which is bad because it loses connections...

Answer 1

You have a few questions in there. For the "add nodes to haproxy without restarting it":

What I do for a similar problem is prepopulate the config file with server names.. e.g. web01, web02 ... web20 even if I only have 5 web servers at the time. Then in my hosts file I map those to the actual ips of the web servers.

To add a new server, you just create an entry for it in the hosts file and it will start passing health checks and get added.

For automated orchestration, it really depends on your environment and you'll probably have to write something custom that fits your needs. There are paid solutions (Scalr comes to mind) to handle orchestration too.

Answer 2

What I do: I have a line in my backend section in haproxy.cfg which says:

# new webservers here

And with a sed script I update haproxy.cfg with something like:

  sed -i -e "/new webservers here/a\    server $ip_address $ip_address:12080 check maxconn 28 weight 100"

And then reload haproxy. Works transparently.

Answer 3

HAProxy has a Runtime API that allows you to do just that dynamically.

Please read the official documentation: Dynamic Configuration HAProxy Runtime API