How to setup ssh tunnel for ipython cluster (ipcluster)

Question

I would like to run an ipython cluster on a ssh-accessible server and on some other machines all of which can connect to each other via ssh. The network is not trusted.

I have tried setting up ssh tunneling using what I understood from the documentation on ipcontroller and ipengine. However the only way that I ended up getting a communication working is by implementing the tunnels manually. Using the instruction from Jean-Francis Roy one would first fix all the necessary ports

controller-host ~ $ ipcontroller --ip=0.0.0.0 --location=127.0.0.1 --port=10101 \
             --HubFactory.hb=10102,10112 \
             --HubFactory.control=10203,10103 \
             --HubFactory.mux=10204,10104 \
             --HubFactory.task=10205,10105

and then one can manually initate the ssh tunnels for the above-fixed ports.

engine-host ~ $ for port in 10101 10102 10112 10103 10104 10105; do\
                    ssh USER@IP_CONTROLLER -f -N -L $port:localhost:$port;\
                done

While this works, I sense from the docs that one of the many option to ipcontroller or ipengine can accomplish this without the need for manual port installation. One quite feasible solution for me could be to open fixed ports to the firewall given that all IPs in our network are fixed and reasonably trusted.

How can I instruct the engines to tunnel to the controller?

Answer 1

Thank you for the copying precious instructions, now the link to the blog is broken! I was able to make it work by editing the corresponding items in the configuration files. First, create a profile :

ipython profile create --parallel --profile=myCluster

Then, edit ~/.ipython/profile_myCluster/ipcontroller_config.py in order to have:

c.HubFactory.control = 10203,10103
c.HubFactory.task = 10205,10105
c.HubFactory.mux = 10204,10104
c.HubFactory.regport = 10101
c.HubFactory.hb = 10102,10112

Now, if the ports are reachable, you just need to launch

ipcluster start --profile=myCluster

For other options, see http://ipython.org/ipython-doc/1/parallel/parallel_process.html#using-ipcluster-in-ssh-mode