Error when building a MapReduce program in Cloudera Hadoop

Question

I get the below error when building a MapReduce file in Hadoop. I am using Cloudera hadoop distribution. testmr_classes is a folder and TestMR.java is the MapReduce file

[cloudera@localhost ~]$ echo `hadoop classpath`
/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/*:/usr/lib/hadoop-0.20-mapreduce/.//*
[cloudera@localhost ~]$

[cloudera@localhost ~]$ javac -classpath `hadoop classpath`:. -d testmr_classes TestMR.java
TestMR.java:32: TestMR.Reduce is not abstract and does not override abstract method reduce(org.apache.hadoop.io.IntWritable,java.util.Iterator<org.apache.hadoop.io.Text>,org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.IntWritable,org.apache.hadoop.io.DoubleWritable>,org.apache.hadoop.mapred.Reporter) in org.apache.hadoop.mapred.Reducer
    public static class Reduce extends MapReduceBase implements Reducer<IntWritable,Text,IntWritable,DoubleWritable>
                  ^
1 error
[cloudera@localhost ~]$

The below are the contents of TestMR.java,

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.util.*;

public class TestMR 
{
    public static class Map extends MapReduceBase implements Mapper<IntWritable,Text,IntWritable,Text>
    {
        private IntWritable key = new IntWritable();
        private Text value = new Text();

        public void map(IntWritable key, Text line, OutputCollector<IntWritable, Text> output, Reporter reporter) throws IOException
        {
            String [] split = line.toString().split(",");
            key.set(Integer.parseInt(split[0]));

            if(split[2] == "Test")
            {
                value.set(split[4] + "," + split[7]);
                output.collect(key, value);
            }
        }
    }

    public static class Reduce extends MapReduceBase implements Reducer<IntWritable,Text,IntWritable,DoubleWritable>
    {
        public void reduce(IntWritable key, Iterable<Text> v, OutputCollector<IntWritable, DoubleWritable> output, Reporter reporter) throws IOException
        {
            Iterator values = v.iterator();
            while(values.hasNext())
            {
                String [] tmp_buf_1 = values.next().toString().split(",");
                String V1 = tmp_buf_1[0];
                String T1 = tmp_buf_1[1];

                if(!values.hasNext())   
                    break;  

                String [] tmp_buf_2 = values.next().toString().split(",");       
                String V2 = tmp_buf_2[0];
                String T2 = tmp_buf_2[1];           

                double dResult = (Double.parseDouble(V2) - Double.parseDouble(V1)) / (Double.parseDouble(T2) - Double.parseDouble(T1));

                output.collect(key, new DoubleWritable(dResult));
            }
        }
    }

    public static void main(String[] args) throws Exception
    {
        JobConf conf = new JobConf(TestMR.class);
        conf.setJobName("TestMapReduce");

        conf.setOutputKeyClass(IntWritable.class);
        conf.setOutputValueClass(DoubleWritable.class);

        conf.setMapperClass(Map.class);
        conf.setCombinerClass(Reduce.class);
        conf.setReducerClass(Reduce.class);

        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(conf, new Path(args[0]));
        FileOutputFormat.setOutputPath(conf, new Path(args[1]));

        JobClient.runJob(conf);
    }
}

This is my first attempt at MapReduce and would be glad to know if I am missing something here.

Answer 1

Look closely at the second arg of your reduce() and the error. You wrote Iterable but it gives you an Iterator.

Answer 2

You need to specify your class path in place of 'hadoop classpath', such as:

javac -classpath /opt/hadoop-1.1.2/hadoop-core-1.1.2.jar:/opt/hadoop-1.1.2/lib/commons-cli-1.2.jar -d testmr_classes TestMR.java 

The exact path depends on where you installed hadoop.