How can I use unix piping across (compute) nodes?

Question

I'm trying to set up a pipeline of processing commands with unix pipes and FIFO:s (named pipes).

I also wanted to send/stream the output of the process to another compute node, which can start working on the stream of data as soon as it arrives, that is, I want to "pipe the stdout over to a process on another machine". How can I do that?

E.g. is it possible to set up a FIFO that will in the background write it's content over to a FIFO on the other compute node, or similar?

Answer 1

You can use netcat. In this minimal example, you can pipe the output of cat to netcat using:

cat local-filename.txt | netcat remote-hostname 1234

Where 1234 represents the TCP port that is going to be used. In the receiving side, you can use:

netcat -l 1234 > filename-on-remote-host.txt

where -l indicates you are setting up a server. This connection will be closed when the originating netcat process finishes. If you need it to keep going and waiting for the next connection, you can use the -k option:

netcat -kl 1234 | some-receiving-command

In any case you can use the abbreviated nc instead of the full netcat:

nc -kl 1234 | some-receiving-command

Answer 2

Yes it is possible, just use ssh for this purpose. The stdin of ssh is sent to the other host. You can use it for example to send data to a different server using tar:

tar cvzf - data | ssh otherhost 'cd /tmp; tar xvzf -'