Setup multinode Hadoop cluster using virtual machines on my laptop

Question

I have a windows 7 laptop and I need to setup hadoop (mutlinode) cluster on it.

I have the following things ready -

• virtual softwares, i.e. virtualbox and vmware player.

• Two virtual machines, i.e.

  Ubuntu - for Hadoop master and

  Ubuntu - for (1X) Hadoop slave

  Has anyone done a setup of such a cluster using Virtual machines on your laptop ?

If yes please help me to install it.

I've searched over google but I am not getting how to configure this multi-node cluster on hadoop using VMs?

How to run two Ubuntu OS on windows 7 using VMware or virtualbox?

Should we use same Ubuntu version VM image or vm images with different versions of Ubuntu linux?

Answer 1

Yes you can use ubuntu two node. I am using five nodes(1 master, 4 datanodes). If you want install multi node in vm ware. Just download ubutnu from this link: http://www.ubuntu.com/download/desktop And install two machine. And install java and openssh. And download shell script for multinode from this link:: https://github.com/tonyreddy/Apache-MultiNode-Insatallation-Shellscript

And try it ..... All the best............

Answer 2

Since you're running Hadoop on your laptop, obviously you're doing it for learning purposes or building POC or functional debugging.

Instead of going through the hassles of installing and setting up Hadoop and related Big-Data softwares, you can simply install a pre-configured pseudo-distributed VM.

Some good options are:

• Cloudera QuickStart VM
• Hortonworks Sandbox

I've been using the Cloudera's VM on my laptop for quite sometime now and it's been working great.

Answer 3

Cloudera and Hortonworks are the fastest way to get it up and running.

Make sure you have enough RAM installed on your laptop for the Operating system already running, else your laptop will restart abruptly often while you use the Virtual machines.

Let me give you an example -

If you are using Windows 10, it needs 3-5GB RAM to be used to work smoothly, This means if you load a Virtual Machine of 5GB size in your RAM, Windows may crash when it does not find enough RAM to operate.

You must upgrade the RAM from 8GB to 12GB or best 16GB for smooth operation of your laptop.

Hope it helps