How can I limit the rate of new outgoing ssh connections when using GNU parallel?

Question

Background: The default setting for MaxStartups in OpenSSH is 10:30:60, and most Linux distributions keep this default. That means there can be only 10 ssh connections at a time that are exchanging keys and authenticating before sshd starts dropping 30% of new incoming connections, and at 60 unauthenticated connections, all new connections will be dropped. Once a connection is set up, it doesn't count against this limit. See e.g. this question.

Problem: I'm using GNU parallel to run some heavy data processing on a large number of backend nodes. I need to access those nodes through a single frontend machine, and I'm using ssh:s ProxyCommand to set up a tunnel to transparently access the backends. However, I'm constantly hitting the maximum unauthenticated connection limit because parallel is spawning more ssh connections than the frontend can authenticate at once.

I've tried to use ControlMaster auto to reuse a single connection to the frontend, but no luck.

Question: How can I limit the rate at which new ssh connections are opened? Could I control how many unauthenticated connections there are open at a given time, and delay new connections until another connection has become authenticated?

Answer 1

I think we need a 'spawn at most this many jobs per second per host' option for GNU Parallel. It would probably make sense to have the default work for hosts with MaxStartups = 10:30:60, fast CPUs, but with 500 ms latency.

Can we discuss it on parallel@gnu.org?

Edit:

--sshdelay was implemented in version 20130122.

Answer 2

Using ControlMaster auto still sounds like the way to go. It shouldn't hit MaxStartups, since it keeps a single connection open (and opens sessions on that connection). In what way didn't it work for you?

Other relevant settings that might prevent ControlMaster from working, given your ProxyCommand setup are ControlPath:

• ControlPath %r@%h:%p - name the socket {user}@{host}:{port}

and ControlPersist:

• ControlPersist yes - persists initial connection (even if closed) until told to quit (-O exit)
• ControlPersist 1h - persist for 1 hour