How to use autoincrement-IDs in Sqoop export

Question

I have a tab-separated textfile in HDFS, and want to export this into a MySQL table.

Since the rows in the textfile do not have numerical ids, how do I export into a table with an ID automatically set during the SQL INSERT (autoincrement)?

If I try to export (id being the last defined attribute in the table), I get

java.util.NoSuchElementException
  at java.util.AbstractList$Itr.next(AbstractList.java:350)
  at entity.__loadFromFields(entity.java:996)

If I take the autogenerated class and modify it to exclude the id-attribute, I get

java.io.IOException: java.sql.SQLException: No value specified for parameter 27

where parameter 27 is 'id'.

Version is Sqoop 1.3.0-cdh3u3

Answer 1

In Sqoop 1.4.1, writing a "null" in the text file field position corresponding to the autoincrement field worked for me. After exported to mySQL you will see an incremented and automatically asigned ID.

Answer 2

As somebody on the Sqoop mailinglist suggested:

• Create a temporary table without the ID
• Sqoop-export into this table
• Copy the rows of this table into the final table (that has the autoincrement ID)

Answer 3

My source table is in HIVE. What works for me is that I add a column called id int, and populate the column as NULL. After sqoop, the mysql will receive insert (id, X, Y) values (null, "x_value, "y_value"). Then mysql knows to populate the id as auto-increment.