Sqoop export from hdfs to SQL Server 2005 using jdts driver fails

Question

I'm trying to export data from hdfs text file to SQL Server using sqoop. When I have more than a couple of rows to insert it throws the following exception:

java.io.IOException: java.sql.SQLException: Incorrect syntax near ','.
    at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:192)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:567)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:675)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.sql.SQLException: Incorrect syntax near ','.
    at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
    at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
    at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
    at net.sourceforge.jtds.jdbc.TdsCore.getMoreResul

I've checked the data for inconsistencies and i can't find anything strange.

I was wondering is the driver even supported?

Answer 1

The problem was being caused with the default way sqoop/jtds group multiple insert statements into 1 using comma separated list of values. This approach is not compatible with sqlserver 2005. To get around it I enabled jdbc-batch insert by providing --batch parameter.

The default way sqoop/jtds group multiple insert statements into 1 is supported with sqlserver 2008.

Also, when I tried the same thing (without the --batch argument) using ms-sql driver everything worked fine. Not sure how ms-sql-driver and sqoop work together to ensure compatibility with sql server 2005.