Portable grid engine platform

Question

I'm looking for a distributed computing platform that can be easily embedded into an application.

I investigated MPI (and some of its implementations) but from what I saw it involves creating an executable for each task and that is launched using mpirun. I don't exclude it, I just want it to be embedded in an application, not as an external executable (if possible).

Do you, SO users, know such a framework (preferably open-source)? Also, in order to make up my mind some usage samples would be nice.

I'm interested in a cross platform solution if possible, as the stations in the LAN where solution is to be deployed is heterogeneous.

Any suggestions and especially code examples are highly appreciated.

Answer 1

Four systems come to mind aside from MPI: AMQP, ZeroMQ, Cilk, and Actors.

AMQP uses message passing to communicate between systems, using a central message broker to handle/distribute the messages. Some implementations guarentee proper delivery to the receiver but are slower (similar to TCP). Some implementations are fire and forget, disallowing guarenteed delivery, but are faster (similar to UDP). Note AMQP is a protocol, not an implementation - popular implementations include ActiveMQ and RabbitMQ.

ZeroMQ is a bit similar to AMQP minus the central message broker. It is basically Actors style I believe, where it is more programmatic, allow you to design your system in an end-to-end manner. It is decentralized (no central message broker) and benchmarks seem to show that it is much faster than AMQP implementations.

I do not have much experience with Cilk other than the fact that it is a well supported project from MIT that is designed to work with C/C++.

Actors are an abstraction to help with both local/distributed concurrency, popularized by Erlang and most recently, the Akka project (Scala, Java). C++ compatible versions that I know of are Theron, Actor-CPP, and libcppa, though I'm not sure which of those support remote actors.

Answer 2

I think Hazelcast is ecatly that kind of thing; embedable distributed computing. But it's in java, not C. But you can call C from Java.