Chef organizing nodes

Question

I have been organizing my managed nodes say I have 12 machines 5 app, 5 db and 2 memcache. I have 3 roles correspondingly. This is Dev environment. I have staging and production environments with large number of machines. So I create chef environments accordingly.

Role: app
    Nodes: 5 app nodes
    Cookbooks: tomcat, java
    override_attributes:  { java_version: 1.6.0 }

and similarly for db and memcache...

If later we decide to upgrade java to 1.7, just update the the role's override_attribute to 1.7 should affect on all 5 nodes.

My question is if I have a situation where only one machine to be left with 1.6.0 and rest all 1.7 what is the best practice.

Options I tried, a. Having an extra if node[fqdn]==HARD_CODED..else condition in cookbook. BUT this situation is only on dev environment. Staging and Production will all have same java version. I don't want to have different cookbooks. b. Override_attributes have a java_version: 1.7 , java_1.6_hosts: "exceptional_host". BUT makes the Role too complex. c. Databags to override. BUT since the configs are in multiple places it makes configs error prone.

I know there is no definite answer for this but I would like to get some ideas on best practices on when applying similar config to a large number of machines and you have a couple of exceptions how to handle them ?

Answer 1

I would go with the option of using a node force_override attribute to define the exception to the rule. Since node force_override attributes are after environment overrides in order of precedence, you should be able to set it on the one node which needs to remain on 1.6, then update the environment to move the other nodes to 1.7.

Answer 2

The problem with chef is that you have so many options... which to choose?

I would recommend leverging version control as much as possible using cookbooks in concert with Berkshelf.

So in version 1.0 of your cookbook instruct that Java6 should be installed:

node.override[:java][:jdk_version] = '6'

include_recipe "java"

But in version 2.0 of your cookbook change this to Java7

node.override[:java][:jdk_version] = '7'

include_recipe "java"

Load both cookbook versions into your chef server and then use environments to control the cookbook version to be used at run-time:

{
  "name": "myapp-dev",
  "json_class": "Chef::Environment",
  "description": "For use by development servers",
   "cookbook_versions": {
    "myapp-java": "= 2.0"
  },
  "chef_type": "environment"
  }
}

{
  "name": "myapp-prod",
  "json_class": "Chef::Environment",
  "description": "For use by production servers",
   "cookbook_versions": {
    "myapp-java": "= 1.0"
  },
  "chef_type": "environment"
  }
}

The advantage of this technique is that it lends itself better to modern CI techniques. You can develop a separate ALM workflow to test and certify the cookbook revisions and then deploy them at run-time. Berkshelf is the tool that makes this possible.