Open source distributed computing/cloud computing frameworks

StackOverflow https://stackoverflow.com/questions/1066749

  •  21-08-2019
  •  | 
  •  

Question

I was wondering if anyone knows of any good Open Source distributed computing projects? I have a computationally intensive program that could benefit from distributed computing (a la SETI@Home, etc.) and want to know if anyone has seen such a thing or will I be developing it from scratch?

Was it helpful?

Solution

Here's one for java and one for c# and here's an open source grid toolkit.

OTHER TIPS

I see that this is over a year old but this is a new and relevant answer

http://openstack.org/

SETI@Home uses BOINC

MPAPI - Parallel and Distributed Applications Framework.

  • Sector 0 Article: http://sector0.dk/?page_id=15.

  • Gives a good overview of the
    framework, architecture and the
    theory behind it.

  • Works on a single machine to 'n'
    machines.

  • Design distributed logic into the
    system.

  • Focuses on message passing to isolate the state that each thread has access to i.e. no shared state only
    messages.

  • Is Open Source =] and is MONO Compatible YAY!

Architecture in a Nutshell

Cluster

  • Single Main Node
  • Controls the cluster
  • Numerous Sub-Nodes (one per machine) which are the work horses of the cluster
  • Single Registration Server - Binds the cluster together by allowing nodes to register / unregister with cluster notifying existing nodes

Communication

  • Node to Node directly. Each worker communicates with others through the node.
  • The messages are not propagated down through the remoting layer unless two workers are on different nodes.

Hadoop if you want to run the machines yourself. Amazon Elastic MapReduce if you want to let others run your workers. Amazon Elastic MapReduce is based on Hadoop.

I have personally used BOINC which is a robust solution, widely used and offer you a great range of possibilities in term of customization.
This is the most complete solution I know. The only problems I had were that it was difficult to use for remote job submission (if you don't have access to the server) and it can be a bit long to setup. But overall it is a very good solution.

If you rather want to implement distributed computing just over a local grid, you can use GridCompute that should be quick to set up and will let you use your application through python scripts.

PS: I am the developer of GridCompute.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top