How to make Fabric ignore offline hosts in the env.hosts list?

Question

This is related to my previous question, but a different one.

I have the following fabfile:

from fabric.api import *

host1 = '192.168.200.181'
offline_host2 = '192.168.200.199'
host3 = '192.168.200.183'

env.hosts = [host1, offline_host2, host3]
env.warn_only = True

def df_h():
    with settings(warn_only=True):
        run("df -h | grep sda3")

And the output is:

[192.168.200.199] run: df -h | grep sda3

Fatal error: Low level socket error connecting to host 192.168.200.199: No route to host

Aborting.

After the execution hits the offline server, it aborts immediately, regardless of the other servers in the env.hosts list.

I have used the env setting "warn_only=True", but maybe I'm using it improperly.

How can I modify this behavior so that it will only prints the error and continue executing?

Answer 1

According to the Fabric documentation on warn_only,

env.warn_only "specifies whether or not to warn, instead of abort, when run/sudo/local encounter error conditions.

This will not help in the case of a server being down, since the failure occurs during the SSH attempt before executing run/sudo/local.

One solution would be to create a function to check if each server is up prior to executing your tasks. Below is the code that I used.

from __future__ import print_function
from fabric.api import run, sudo, local, env
import paramiko
import socket

host1 = '192.168.200.181'
offline_host2 = '192.168.200.199'
host3 = '192.168.200.183'

env.hosts = [host1, offline_host2, host3]

def df_h():
    if _is_host_up(env.host, int(env.port)) is True:
        run("df -h | grep sda1")


def _is_host_up(host, port):
    # Set the timeout
    original_timeout = socket.getdefaulttimeout()
    new_timeout = 3
    socket.setdefaulttimeout(new_timeout)
    host_status = False
    try:
        transport = paramiko.Transport((host, port))
        host_status = True
    except:
        print('***Warning*** Host {host} on port {port} is down.'.format(
            host=host, port=port)
        )
    socket.setdefaulttimeout(original_timeout)
    return host_status

Answer 2

As of version 1.4 Fabric has a --skip-bad-hosts option that can be set from the command line, or by setting the variable in your fab file.

env.skip_bad_hosts = True

Documentation for the option is here: http://docs.fabfile.org/en/latest/usage/fab.html#cmdoption--skip-bad-hosts

Don't forget to explicitly set the timeout value also.

Answer 3

You're not using it improperly. You can even just provide --warn-only=true on the command line. It's the documented method suggested by the development team.

Answer 4

Based on Matthew's answer, I came up with a decorator that accomplishes just that:

from __future__ import with_statement
from paramiko import Transport
from socket import getdefaulttimeout, setdefaulttimeout
from fabric.api import run, cd, env, roles


roledefs = {
    'greece': [
        'alpha',
        'beta'
    ],
    'arabia': [
        'kha',
        'saad'
    ]
}

env.roledefs = roledefs


def if_host_offline_ignore(fn):
    def wrapped():
        original_timeout = getdefaulttimeout()
        setdefaulttimeout(3)
        try:
            Transport((env.host, int(env.port)))
            return fn()
        except:
            print "The following host appears to be offline: " + env.host
        setdefaulttimeout(original_timeout)
    return wrapped


@roles('greece')
@if_host_offline_ignore
def hello_greece():
    with cd("/tmp"):
        run("touch hello_greece")


@roles('arabia')
@if_host_offline_ignore
def hello_arabia():
    with cd("/tmp"):
        run("touch hello_arabia")

It is especially useful when you have multiple hosts and roles.