Question

i am totally new to the concept of totalorderpartitioner,i have applied this concept but im not successful to produce global sort. this is my input record

676576
7489768576
689576857867857
685768578678578675
765897685789675879679587
1
5
6
7
8
9
0
2
3
5
6
9

this is my mapper

public void map(LongWritable key, Text value,
            OutputCollector<NullWritable, Text> outputCollector, Reporter reporter) throws IOException {
        // TODO Auto-generated method stub
        outputCollector.collect(NullWritable.get(),value);

    }

this is my reducer

public void reduce(NullWritable key, Iterator<Text> values,
            OutputCollector<NullWritable, Text> outputCollector, Reporter reporter) throws IOException {
        // TODO Auto-generated method stub
        while (values.hasNext()) {
            Text text = (Text) values.next();
            outputCollector.collect(key,text);

        }

    }

this is my job related code

JobConf jobConf = new JobConf();
jobConf.setMapperClass(TotalOrderMapper.class);
jobConf.setReducerClass(TotalOrderReducer.class);
jobConf.setMapOutputKeyClass(NullWritable.class);
jobConf.setMapOutputValueClass(Text.class);
jobConf.setOutputKeyClass(NullWritable.class);
jobConf.setOutputValueClass(Text.class);
jobConf.setPartitionerClass(TotalOrderPartitioner.class);
jobConf.setInputFormat(TextInputFormat.class);
jobConf.setOutputFormat(TextOutputFormat.class);
FileInputFormat.addInputPath(jobConf, new Path("hdfs://localhost:9000/totalorderset.txt"));
FileOutputFormat.setOutputPath(jobConf, new Path("hdfs://localhost:9000//sortedRecords5.txt"));
Path pa = new Path("hdfs://localhost:9000//partitionfile","_partitions.lst");
TotalOrderPartitioner.setPartitionFile(jobConf,
        pa);
InputSampler.writePartitionFile(jobConf,
        new InputSampler.RandomSampler(1,1));
JobClient.runJob(jobConf);

but the records are not getting sorted.this is my output

676576
7489768576
689576857867857
685768578678578675
765897685789675879679587
1
5
6
7
8
9
0
2
3
5
6
9

i dont know where iam going wrong.could anyone help me to sort out the problem?and could anyone tell me how the inputsampling works exactly here.thanks in advance

Was it helpful?

Solution

Mapreduce sorts on key, since you are sorting on nullwritable you are doing no sorting at all:

outputCollector.collect(NullWritable.get(),value);

Your map output keys should be your inputvalues!

outputCollector.collect(value, NullWritable.get());

Can you give this a try and let us know if it does the trick?

Second remark: Use IntWritable instead of Text otherwise the sorting will be lexicographically!

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top