Hadoop Serializer Not Found Exception

Question

I have a job whose output format is SequenceFileOuputFormat.

I set the output key and value class like this:

conf.setOutputKeyClass(IntWritable.class);
conf.setOutputValueClass(SplitInfo.class);

The SplitInfo class implements Serializable,Writable

I set the io.serializations property as follows:

conf.set("io.serializations","org.apache.hadoop.io.serializer.JavaSerialization," 
+ "org.apache.hadoop.io.serializer.WritableSerialization");

However, on the reducer side I get this error, telling me that Hadoop couldn't find a serializer:

    java.lang.NullPointerException
at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:961)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:892)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:393)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:61)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:569)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:638)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)

Can anyone help, please ?

Answer 1

The problem was that I was making a stupid mistake: I was not updating a jar. So, basically SplitInfo was not implementing the Writable interface in the old (in use) jar.

As a general observation: the error specified in the OP has as underlying cause the fact that HADOOP can't find a Serializer for a specific type which you're trying to serialize (being directly or indirectly, e.g. by using that type as an output key/value). Hadoop cannot find a Serilizer for one of the 2 reasons:

1. your type is not serializable (i.e. it doesn't implement Writable or Serializable)
2. There is no Serializer available to Hadoop for the type of serialization your type implements (e.g.: your type implements Writable but hadoop for one reason or another cannot use the org.apache.hadoop.io.serializer.WritableSerialization class)

Answer 2

I think you're trying to do something you don't need to. Your output value only needs to implement the Writable interface and you should just set the output format.

conf.setOutputFormatClass(SequenceFileOutputFormat.class);

You only use the "io.serializations" configuration if you want to use a different serialization framework, which it doesn't look like you need.