Is there any way to create an autoscaling Hadoop Hbase cluster

Question

I used aws autoscaling for my webserver and Apache whirr for hadoop habse cluster. Do Apache whirr has an autoscaling property.

Answer 1

You can use Apache Mesos which a container based framework. Mesos works on container(e.g. dockers to start the services.) In mesos you can configure number of instances you need, configure resources, scale up/down etc. It will also take care of updating maintaining state of the process. This has been recently published hbaseconf 2015 where adobe has started using mesos for auto scaling in their production. You can also use google kuberneetes or marathon for managing the cluster and configurations. I have personally tried this on my local cluster. For more information to get started please follow this link

Basic introduction and features of the mesos for quick reference [source: apache site]

What is Mesos? A distributed systems kernel

Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments.

Project Features

1. Scalability to 10,000s of nodes
2. Fault-tolerant replicated master and slaves using ZooKeeper
3. Support for Docker containers Native isolation between tasks with Linux Containers
4. Multi-resource scheduling (memory, CPU, disk, and ports)
5. Java, Python and C++ APIs for developing new parallel applications
6. Web UI for viewing cluster state