Parallel processing middleware

Question

For relatively coarse-grained parallel processing (data transfers can be done in 100 Mbit Ethernet network), which middleware that is available for Windows and Linux do you consider best? CORBA? MPI1? MPI2? XML-RPC/SOA/WSDL? Custom network protocols? Java RPC? Python RPC? Others?

Answer 1

PVM was very useful to me for a small (10 machine, 40 processor) cluster with a GigE switch. This was a couple years ago, but should still be viable, though PVM does not seem to be evolving any further. I mention it to add it to you list of candidates. I suspect MPI would be a more popular choice, but I liked PVM.

Answer 2

You're asking about middleware, but then specifying a series of messaging protocols. If you're looking for messaging protocols, the Thrift framework from Facebook is definitely something that works under very high load and cluster size:

Thrift

If you're looking for some other component of middleware, please clarify.

Answer 3

Or may be Terracotta? Or may be Hadoop?