hadoop reduce function is not excuted?
문제
I am running the mapreduce program, but I realize that although I write a reduce function, the mapreduce does not exceute it. I then try to add @Override
, the error says the reduce function does not override any super methods.
Job configuration information is as below
preProcess.setJobName("GeneProcessing program for 100 SNP");
preProcess.setMapperClass(PreprocessMapper.class);
preProcess.setReducerClass(PreprocessReducer.class);
preProcess.setInputFormatClass(TextInputFormat.class);
//setInputFormat(TextInputFormat.class);
preProcess.setOutputFormatClass(TextOutputFormat.class);
preProcess.setMapOutputKeyClass(Text.class);
preProcess.setMapOutputValueClass(Text.class);
preProcess.setOutputKeyClass(NullWritable.class);
preProcess.setOutputValueClass(Text.class);
//preProcess.setNumMapTasks(4);
preProcess.setNumReduceTasks(4);
FileInputFormat.setInputPaths(preProcess, preprocessInputPath);
FileOutputFormat.setOutputPath(preProcess, preprocessOutputPath);
//JobClient.runJob(preProcess);
preProcess.waitForCompletion(true);
Reduce code is as below
import java.io.*;
import java.util.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.Reducer;
public class PreprocessReducer extends Reducer<Text, Text, NullWritable, Text>
{
public Text newKey = new Text("");
//no enter the reduce task
@Override
protected void reduce(Text key, Iterator<Text> values, Context output)
throws IOException, InterruptedException
{
do something here
}
}
해결책
Your signature for the reduce method is wrong - the second argument should be Iterable<Text>
rather than Iterator<Text>
:
@Override
protected void reduce(Text arg0, Iterable<Text> arg1,
Context arg2)
throws IOException, InterruptedException {
}
What IDE are you using? Eclipse has a Source -> Override / Implement method menu option which makes getting the signature correct much easier. I'm sure Netbeans has a similar function.
제휴하지 않습니다 StackOverflow