Preventing single entry point in configuration management master/agent setup

Question

I'm researching configuration management software like Puppet. My primary concern is preventing a single entry point to all of our internal servers. Take this scenario for example.

Somehow, access is gained into the master configuration server. From there a user would then be able to gain relatively easy access to manipulate or ultimately gain access to other servers controlled by the master.

The primary goal is to prevent a single point entry into the network, even if said master configuration is not available to the public internet.

tl;dr How can I prevent single point access to all other servers in a master/agent configuration management setup?

Answer 1

If you are thinking about delegating the task of defining the Puppet rules to other people (eg. technician), you can have create a Puppet master (Master A), have a test machine connected to Master A, then make them commit the code to Git or SVN.

You control a second Puppet master (Master B), which you pull the code from Git or SVN. All your machine in network connect to Master B. Once you are happy with the code, you can ask Puppet to push it to all your machine.

This way, access to all machine configuration only on Master B, which only you handle and access.

Answer 2

You can't with the default configuration.

The way Puppet is designed is to have agents to contact the master. Even if you are behind multiple firewalls you need to allow the agents to enter into your internal network. Even if you routinelly allow connections from the DMZ to the internal network, you may still need to manage machines in the open internet. What Puppet requires is to open your internal network to the open internet.

The risk of this client-pull design is that if you can hack into a machine with an agent you can contact the master, and if the master have any vulnerability you can hack into it and from them you can control all machines with agents, plus you can mount an attack into your internal network. So, if a vulnerability is exploited in the Puppet master communication channel with the agents, then Puppet becomes an attack vector (a huge one, as you maybe managing all your infrastructure with it, and you have allowed access from outside to your LAN).

With a master-push design this could be minimized, as the master would be one single point to protect and would be inside the safe internal network, with connections only going from inside to outside.

There is a pending feature request (4 years old!) in PuppetLabs (http://projects.puppetlabs.com/issues/2045) titled Push functionality in puppetmaster to clients. Reading the comments on that feature request and finding things like the following comment makes me wonder if the Puppet developers really understant what is the problem:

Ultimately, it isn’t all that high a priority, either – almost every risk that opening the port to the master exposes is also exposed by having the master reach out and contact the client. There is little or no change in actual risk to the model proposed.

However, while the developers realize the problem, others are designing their own solutions (like https://github.com/tomas-edwardsson/puppet-push).

Update: I've found a presentation by Bernd Strößenreuther titled Best practices on how to turn Your environment into a Puppet managed environment available as PDF at http://stroessenreuther.info/pub/Puppet_getting_started.pdf

He suggest to establish ssh connections from the master to the agents and open a reverse tunnel so that the agents can connect to the master. These connections could be started in a cron job periodically. In this way you don't have to open your internal network for incoming connections yet the agents have access to the master data.

Now, regarding the pull mechanism, it may seem like a bad design but actually it is essential to allow very automated environments to work. For example, in an elastic network (like EC2 with autoscaling) where servers are started and halted automatically, the servers need to be able to configure themselves right away, so they boot up and the first thing they do is contact the master for an updated configuration. That would be harder if you have to push the configuration to each server periodically, because they would need to wait for the master (seconds, minutes, or hours; that is unacceptable in some applications).