TCP Socket communication between processes on Heroku worker dyno

Question

I'd like to know how to communicate between processes on a Heroku worker dyno.

We want a Resque worker to read off a queue and send the data to another process running on the same dyno. The "other process" is an off-the-shelf piece of software that usually uses TCP sockets (port xyz) to listen for commands. It is set up to run as a background process before the Resque worker starts.

However, when we try to connect locally to that TCP socket, we get nowhere.

Our Rake task for setting up the queue does this:

task "resque:setup" do
  # First launch our listener process in the background
  `./some_process_that_listens_on_port_12345 &`

  # Now get our queue worker ready, set up Redis backing store
  port = 12345
  ENV['QUEUE'] = '*'  
  ENV['PORT'] = port.to_s
  Resque.redis = ENV['REDISTOGO_URL']

  # Start working from the queue
  WorkerClass.enqueue
end

And that works -- our listener process runs, and Resque tries to process queued tasks. However, the Resque jobs fail because they can't connect to localhost:12345 (specifically, Errno::ECONNREFUSED).

Possibly, Heroku is blocking TCP socket communication on the same dyno. Is there a way around this?

I tried to take the "code" out of the situation and just executed on the command line (after the server process claims that it is properly bound to 12345):

nc localhost 12345 -w 1 </dev/null

But this does not connect either.

We are currently investigating changing the client/server code to use UNIXSocket on both sides as opposed to TCPSocket, but as it's an off-the-shelf piece of software, we'd rather avoid our own fork if possible.

Answer 1

Have you tried Fifo?

http://www.gnu.org/software/libc/manual/html_node/FIFO-Special-Files.html#FIFO-Special-Files

Answer 2

Use message queue Heroku add-ons ...,

like IronMQ for exsample

Answer 3

Reading your question, you've answered your own question, you cannot connect to localhost 12345.

This way of setting up your processes is a strange one as your running two processes within one Heroku dyno which removes a lot of the benefits of Heroku, i.e independant process scaling, isolation and clean depenedency declaration and isolation.

I would strongly recommend running this as two seperate processes that interact via a third party backing service.

Answer 4

Heroku only lets you listen in a given port ($PORT) per dyno, I think.

I see two solutions here:

• Use Redis as a communication middleware, so the worker would write on Redis again and the listener process, instead of listening in a port would be querying redis for new jobs.

• Get another heroku dyno (or better, a complete different application) and launch there the listening process (on $PORT) and communicate both applications

Answer 5

@makdad, is the "3rd party software" written in Ruby? If so, I would run it with a monkey patch which fakes out TCPSocket or whatever class it is using to access the TCP socket. Put the monkey patch in a file of its own, which will only be required by the Ruby process which is running the 3rd party software. The monkey patch could even read data directly from the queue, and make TCPSocket behave as if that data had been received.

Yes, it's not very elegant, and I'm sure there may be a better way to do it, but when are you trying to get a job done (not spend days doing research), sometimes you just have to bite the bullet and do something which is ugly, but works. Whatever solution you choose, make sure to document it for those who work on the project later.