Executing long task on a remote server with python-rq

Question

I have written some code which takes a long time to execute (2-3 days) and i want to push it on the server to be executed there. The code is rich of classes and functions interacting with each other but in the end the whole code execution is done through a single function (test2) which will make it work. I have found out that a solution for me might be a task queue and as i do not need to execute multiple tasks at the same time i've found that RQ might suit my needs.

#action_test.py

import action2

def test1():
    fl = action2.FollowersList()
    mech = action2.Mechanics()
    manager = action2.Manager()
    manager.launch(mech,fl)
    for i in range(0,10):
        manager.iterate(mech,fl)

def test2():
    messageList = []
    fl = action2.FollowersList()
    mech = action2.Mechanics()
    manager = action2.Manager()
    manager.launch(mech,fl)
    for i in range(0,2000):
        message = manager.iterate(mech,fl)
        messageList.append(message)
    return messageList

I've set up Reddis on remote server. Run it in a daemon mode. Than i wrote a simple module which should just put my test2 function in a queue.

#app.py

from rq import Connection, Queue
from redis import Redis
from action_test import test2

def main():
    # Tell RQ what Redis connection to use
    redis_conn = Redis()
    q = Queue(connection=redis_conn)  # no args implies the default queue

    # Delay calculation of the multiplication
    job = q.enqueue(test2, timeout = 259200)
    print job.result   # => None

if __name__ == '__main__':
    main()

Then i have encountered an issue: on a python-rq docs webpage (http://python-rq.org/docs/workers/) the described way to launch a worker is to execute

$ rqworker

from the shell. But this worker starts not as a daemon and thus as i am connecting to this remote server where my app is set up over ssh, if my ssh connection goes down the worker goes down as well and this is not quite a behavior i would like it to have. Maintaining the ssh connection for 2-3 days while my code executes denies the whole logic beyond using python-rq in my case. Are there any ways around this issue? Perhaps a python-rq worker should be launched not from the shell to be daemonized?

Answer 1

You can run the worker in background (&) and send the output to a textfile (nohup):

nohup rqworker &

Bydefault this will write the output to a file nohup.out within the same directory (or $HOME/nohup.out if that's not permitted). You can now close the ssh connection.

With default settings, rq will write a lot to this file but --quiet helps:

nohup rqworker --quiet &

See man nohup and how to start jobs in background.

Answer 2

As far as I can tell, you don't need python-rq at all. A message queue is typically used for communication in a distributed system, which is way overkill for somebody that just wants to run a long-running script on a remote server. You could skip the rqworker and run your script directly. Add if __name__=='__main__': test2() to your action_test.py script, then you can run the script from the command line:

python test_action.py

As you point out, when you close your SSH session, your script also gets killed. That's because of some Unix/Linux magic called "signals": the system sends a "hangup" signal to your job, also known as SIGHUP. Martin mentioned nohup already and that's a workable answer (e.g. nohup python test_action.py) but it's a bit complicated for a beginner who just wants to run a long script.

A simpler solution for you is to use screen. Screen creates a virtual terminal that lets you disconnect from a shell and reconnect later. If you have screen installed, just run screen and it will create a new virtual terminal for you. Now run your program as normal: python test_action.py. Once it gets started, type Ctrl+A and then D to disconnect. Now you can disconnect your SSH session but the job will keep running in the virtual terminal as if nothing happened. Later on, you can SSH back to the server and type screen -r to resume using that terminal.

More info here: https://unix.stackexchange.com/questions/24658/nohup-vs-screen

Answer 3

You can use the supervisord package for this. It helps to set up a deamon process for running task in background. And it is also quite easy to configure.