Inter-application communication via the cloud?

Question

I have a very general question and am looking for a general point to the right direction. Say if I wanted to build a program that could be run on several computers but wanted each program to work with the other, how could I achieve the communication layer between them?

Ex. Machine A in chicago IL is running

Machine B is running in Houston TX

Machine C is running in San Diego CA

They all serve one function but work on it as a team to get it done quicker.

Machine A turns on and is ready to go but needs to check if machine B and C are ready. Once all machines are on and check with each other then they start. There is some type of que that they all pick jobs out of and complete. There for they have to know not to do a job that another computer/thread is already working on.

I have never dealt with inter-computer communication when it comes to pairing them up as a team so I really have no idea where to begin. I was thinking about having a cloud database where they can all check in and keep tabs on which machine is doing what but that seems very clunky. I'm sure there is a better way to do it.

Answer 1

You want two different things here. You want a pub/sub or topic (http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) that they all check into on a regular basis... If anyone doesn't check in every x number of minutes/seconds they stop work (assuming you want the to stop, I'd have to vet that requirement). You also want a simple message queue (http://en.wikipedia.org/wiki/Message_queue) to pick work off.

I'd still double-check whether you really need to have them all running before they start work. I wouldn't geographically separate them. If they are just stages in a pipeline, make several queues and when they run out of work they sleep. This is the advantage of message based architecture. Decoupling potentially high-lag processes from each other. Try and keep highly dependent work co-located.