Agent Framework on Hadoop

Question

I was thinking of writing a simple agent based framework using Hadoop. This is an idea I have been toying around for quite a while now (but have not put a lot of deep thinking into).

The target application I want to use it for is Genetic Programming. The Idea is that I would spawn a lot of agents, have them perform a task, evaluate their genetic fitness and then breed them, and then repeat the whole exercise.

I wanted to know if folks had any immediate thoughts/critique of doing this using Hadoop.

Thanks in advance.

Answer 1

I think you should look into YARN of hadoop 2.0. It enables you to run any arbitrary logic not restricting you to MR paradigm

Answer 2

It depends on how your agents really fit map/reduce approach. If results produced by agents will be consumed in reduce phase and new jobs will be initiated - you're ok with Hadoop. Buf if your agents depend on results of each other - you probably want to try out something like Storm ( https://github.com/nathanmarz/storm/wiki )