fabric: run() hangs without any errors

Question

from fabric.api import env
from fabric.api import run

class FabricSupport:
    def __init__ (self):
        pass

    def run(self, host, command):
        env.key_filename = 'C:/.ssh/dev.pub'
        env.host_string = "%s:%s" % (host, 22)
        run(command, timeout=5)

fab = FabricSupport()

def execute(host, txt):
    stdout = ""
    try:
        stdout = fab.run(host, txt)
    except Exception, e:
        stdout = str(e)
    return stdout

result = execute("23.23.23.23", "uname -a")

it just outputs the following and stays like that indefinitely. timeout doesn't seem to be working. It would be great if it was more verbose so I could investigate what is going on.

[12.45.241.11:22] run: uname -a

Answer 1

I'm not sure what your end goal is with this code snippet. If you're trying to isolate the effects of modifying env by having it in a class, that's not going to work. It's a module level global in fabric and no matter where you modify it (by assignment like you're doing here) it's going to apply for any subsequent code. What I think you probably really want is this:

from fabric.api import run, execute, task, settings

@task
def custom_runner(command):
    """Runs command with custom key_filename"""
    with settings(warn_only=True, key_filename=r"C:\.ssh\dev.pub"):
        result = run(command, timeout=5)
    return result

result = execute(custom_runner, "uname -a", host="23.23.23.23")

Keep in mind that this can't be used by the "fab" command line utility as-is because it's executing that code (the result = execute...) as soon as fab loads the fabfile. This would be strictly library use.

You can use roledefs to do groups of hosts like this. The result object will be a dictionary where each host is a key and the corresponding values are the return object of the task. You almost never want to use a bare Exception clause, Fabric has its own ways to prevent an exception from causing your program to terminate.

Answer 2

I encountered the same problem today with an Ubuntu host. My problem was the active shell setting:

env.shell = /bin/sh -l -c

With this shell the script would not cleanly execute the remote command and hangup the ssh connection. The control remained inside the ssh shell, causing a timeout in fabric.

With the setting:

env.shell = "/bin/bash -c"

everything was OK.

General advice: check the shell and parameters according to your host OS.

Answer 3

I came across the same hanging error this week when I need to run a job taking long time on remote machine.

I resolved it by specifying 'keepalive' option in fabric.api.env config. This config will ask fabric to send keepalive message every a few minutes.

http://docs.fabfile.org/en/1.14/usage/env.html#keepalive

I set the value to be 30 and it doesn't hang on a job running for 3 hours.