YARN as a SOA framework

Question

We are considering building a service oriented architecture on top of YARN. We have different application types - some would work in Storm like streaming mode (where we connect to the running service), some in batch processing mode (when the app is started on every request).

Moreover applications might need to communicate to each other often which would require a lot of internal traffic between different applications within YARN. We want to use as well the caching of different applications, so whenever the request with the same data goes to the same app we can return cached responses.

Is YARN a good or bad solution as a basis for SOA framework? Is Yarn just a autoscaling/deployment-like tool or would it be a good fit for SOA? Would it be fast enough to do this with YARN?

Answer 1

The way I see it YARN is pushing Hadoop form being a distributed file system to a distributed OS. There are a lot of SOA-ish infrastructures that are being built or migrating to YARN (Storm, Samza) that are compelling servicehosts. You can also at weave from continuuity, that will help you host additional types of services.

to specifically address you q. - YARN is a good basis for SOA framework, it is more than a autoscaling it is a resource management and hosting framework and it is fast enough (esp. if you use one of the already developed infrastructures that are built on top of it)