How can I secure memcached/beanstalkd in a hostile cloud environment?

Question

Here's how my servers (in Amazon EC2) would look like:

        Server 1                    Server 2                  Server 3
__________________________     _____________________      _______________
|  Cloud Monitor Daemon  |    |  Memcached daemon  |      |  beanstalkd  |
|                        |    |  Memcached daemon  |      ________________
| "Hostile" user process | /  |  Memcached daemon  |
| "Hostile" user process | /  |  Memcached daemon  |
| "Hostile" user process | /  |  Memcached daemon  |
| "Hostile" user process | /   ______________________
| "Hostile" user process | /
__________________________

There's multiple user processes on one server. Each user then has their own memcached instance running on a (separate) server (with many other memcached instances). Without any sort of security (as it is by default), user process B could guess the port of the memcached instance of user A and access it. How can I secure this system so that user C could only access memcached instance C and no other (even though the memcached instances are all on the same server)? My user should not have to do anything to make use of the security (just continue connecting to the memcached port as usual), it should all happen automatically by the system.

Also, the Cloud Monitor Daemon on the server along with the "hostile" user processes needs to be able to access a remote beanstalkd server. Beanstalkd has no authentication either, so if my Monitor Daemon can access beanstalkd, so can the "hostile" user processes, and I don't want that. How can I secure this part?

Answer 1

I mentioned some tips on securing memcached in a blog post recently. For your case, SASL will probably help a lot.

I don't know if beanstalk ever got SASL support, but that's kind of a different app.

You could build a VPN or enable IPSEC to control access to all services on all machines at the node level, of course.

Answer 2

You can start beanstalkd on Server3 local IP (127.0.0.1) and then use SSH Tunnels from the Server 1 to Server 3. Combine it with inetd and ssh-keys to be failsafe.

Answer 3

A couple weeks ago Amazon has announced the Amazon VPC (Virtual Private Cloud) which we are using to secure memcached and beanstalkd.

Works great! Seriously reccomend it; one less overhead to have to deal with ourselves.

Answer 4

I ended up going with plain old iptables. Allows me to do per-uid rules and is very easy to configure programatically. Most importantly, the users don't need to be involved in the process, they can continue using the standard protocols and not have to deal with authentication, and iptables will drop any "naughty" packets that are going where they shouldn't.