High availability deployment with Ansible

Question

I'm using Ansible to deploy pairs of NGinx/Tomcat instances and I'm trying to improve availability during deployment.

A logical instance is 1 NGinx + 1 Tomcat: I have 4 logical instances spread over 2 distant locations (see hosts file bellow).

I launch one playbook called deploy.xml which looks like this:

- hosts: ngx-servers
  pre_tasks:
    - include: tasks/remove-server.yml
  roles:
    - role: ngx-server

- hosts: app-servers
  roles:
    - role: app-server

- hosts: ngx-servers
  tasks:
    - include: tasks/add-server.yml

What I want is to deploy 50% of my 4 logical instances before deploying the others (and stop everything if something goes wrong). One solution could be targeting montigny-app-servers/montigny-ngx-servers first (instead of app-servers/ngx-servers) and then the second location but I would need to duplicate playbook content (and so on if I need to add other server locations).

Any idea to make it properly ?

Here is my hosts file:

#
# Serveurs d'application
#

# Montigny
[montigny-app-servers]
mo-app-server-1 ansible_ssh_host=1y.domain.fr ansible_ssh_user=devops
mo-app-server-2 ansible_ssh_host=2y.domain.fr ansible_ssh_user=devops

# Bievre
[bievre-app-servers]
bi-app-server-1 ansible_ssh_host=1b.domain.fr ansible_ssh_user=devops
bi-app-server-2 ansible_ssh_host=2b.domain.fr ansible_ssh_user=devops

# Tous
[app-servers:children]
montigny-app-servers
bievre-app-servers

#
# Serveurs NGinx
#

# Montigny
[montigny-ngx-servers]
mo-ngx-server-1 ansible_ssh_host=1y.domain.fr ansible_ssh_user=devops
mo-ngx-server-2 ansible_ssh_host=2y.domain.fr ansible_ssh_user=devops

# Bievre
[bievre-ngx-servers]
bi-ngx-server-1 ansible_ssh_host=1b.domain.fr ansible_ssh_user=devops
bi-ngx-server-2 ansible_ssh_host=2b.domain.fr ansible_ssh_user=devops

# Tous
[ngx-servers:children]
montigny-ngx-servers
bievre-ngx-servers

Answer 1

One solution could be targeting montigny-app-servers/montigny-ngx-servers first (instead of app-servers/ngx-servers) and then the second location

This is what we do, couple changes:

Add geolocated groups:

[Montigny-servers:children]
montigny-app-servers
montigny-ngx-servers

[bievre-servers:children]
bievre-app-servers
bievre-ngx-servers

Then you can execute you playbook on the subsets with the options -l/--limnit

$ ansible-playbooks -i inventory site.yml -l Montigny-servers && \
  ansible-playbooks -i inventory site.yml -l bievre-servers

this way it will first run you logical instances from Montigny, and if everything went fine (exit 0), it will run the playbook for bievre instances

Find more exemples: http://docs.ansible.com/playbooks_best_practices.html#what-this-organization-enables-examples

Answer 2

I agree with the idea of putting your hosts in separate groups so that you can run commands on first one group and then the other.

You should be able to put that in one playbook though - if you create the fundamental actions in an include script, you can then create a playbook that will have multiple '- hosts' sections, each of which will call the include script.

That would look something like:

---
- hosts: firstgroupname
  tasks:
    - include: pathandnameofincludescript.yaml
- hosts: secondgroupname
  tasks:
    - include: pathandnameofincludescript.yaml

and your include script would look something like:

---
- name: action 1
  command: echo hello world
- name: action 2
  command: echo hello again

Note the indentation difference for include scripts - its very important! YAML can be annoying at times....

Answer 3

Okay, here is what I think you actually want.

You can use the serial keyword to say how many hosts ansible should manage at a single time in a play. So in the example below, ansible would only run the plays on two hosts at a time.

By default, anisble will keep running the plays even if the first two machines fail, so you can use the max_fail_percentage keyword to define the percentage of failures acceptable before stopping. In the example below, Ansible will stop if 1 percent of machines failed.

Source: http://docs.ansible.com/playbooks_delegation.html#rolling-update-batch-size

- hosts: app-servers`
  serial: 2 
  max_fail_percentage: 1
  roles: 
   - role: deploy-env

And then in your deploy-env role have:

- name: remove from LB
  include: remove-server.yml
  delegate_to: paired_nginx_box

  /* app-server tasks */

- name: add to LB
  include: add-server.yml
  delegate_to: paired_nginx_box