SerDe problems with Hive 0.12 and Hadoop 2.2.0-cdh5.0.0-beta2

Question

The title is a bit weird as I'm having difficulties narrowing down the problem. I used my solution on Hadoop 2.0.0-cdh4.4.0 and hive 0.10 without issues.

• I can't create a table using this SerDe: https://github.com/rcongiu/Hive-JSON-Serde

first try:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.hadoop.hive.serde2.objectinspector.primitive.AbstractPrimitiveJavaObjectInspector.<init>(Lorg/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils$PrimitiveTypeEntry;)V

second try:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Could not initialize class org.openx.data.jsonserde.objectinspector.JsonObjectInspectorFactory

• I can create a table with this SerDe: https://github.com/cloudera/cdh-twitter-example

• I create an external table with tweets from flume. I can't do "SELECT * FROM tweets;"

  FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException: java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDe

• I can do SELECT id, text FROM tweets;

• I can do a SELECT COUNT(*) FROM tweets;

• I can't self join this table:

  Execution log at: /tmp/jochen.debie/jochen.debie_20140311121313_164611a9-b0d8-4e53-9bda-f9f7ac342aaf.log 2014-03-11 12:13:30 Starting to launch local task to process map join; maximum memory = 257294336 Execution failed with exit status: 2 Obtaining error information

  Task failed! Task ID: Stage-5

mentioned execution log:

2014-03-11 12:13:30,331 ERROR mr.MapredLocalTask (MapredLocalTask.java:executeFromChildJVM(324)) - Hive Runtime Error: Map local work failed
org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException: java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDe

Does anyone know how to fix this or at least show me where the problem is?

EDIT: Can it be a problem that I built the serde on a Hadoop 2.0.0-cdh4.4.0 and hive 0.10?

Answer 1

From what I've seen, Hive-.11+ has a bug in join with custom SerDe.

https://github.com/Esri/gis-tools-for-hadoop/issues/9

You might try the workaround of copying the JAR file containing the SerDe class, to $HIVE_HOME/lib .

(I see in your question you got ClassNotFoundException both in join and in other cases; so far the times I have encountered such were all with join.)

[Edit] Another workaround is to use HADOOP_CLASSPATH:

env HADOOP_CLASSPATH=some.jar:other.jar hive ...

[Edit] The work around applies to Hive versions 0.11 and 0.12; then 0.13 and above contain the fix for HIVE-6670.