Solr DIH Error: Unable to read: dataimport.properties

Question

After installing Solr 4, I did a full import using DIH and I get the following error in the error logs:

Catalina.out

22-Nov-2012 08:36:21 org.apache.solr.handler.dataimport.DataImporter maybeReloadConfiguration
INFO: Loading DIH Configuration: /opt/solr/collection1/conf/data-config.xml
22-Nov-2012 08:36:21 org.apache.solr.handler.dataimport.config.ConfigParseUtil verifyWithSchema
INFO: id is a required field in SolrSchema . But not found in DataConfig
22-Nov-2012 08:36:21 org.apache.solr.handler.dataimport.config.ConfigParseUtil verifyWithSchema
INFO: latlng is a required field in SolrSchema . But not found in DataConfig
22-Nov-2012 08:36:21 org.apache.solr.handler.dataimport.DataImporter loadDataConfig
INFO: Data Configuration loaded successfully
22-Nov-2012 08:36:21 org.apache.solr.handler.dataimport.DataImporter doFullImport
INFO: Starting Full Import
22-Nov-2012 08:36:21 org.apache.solr.core.SolrCore execute
INFO: [collection1] webapp=/solr path=/dataimport params={command=full-import} status=0 QTime=5 
22-Nov-2012 08:36:21 org.apache.solr.handler.dataimport.SimplePropertiesWriter readIndexerProperties
WARNING: Unable to read: dataimport.properties
22-Nov-2012 08:36:21 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Creating a connection for entity listings with URL: jdbc:mysql://localhost/mysite
22-Nov-2012 08:36:21 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Time taken for getConnection(): 15
22-Nov-2012 08:36:25 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] webapp=/solr path=/dataimport params={command=full-import} status=0 QTime=5 {deleteByQuery=*:*} 0 5
22-Nov-2012 08:36:25 org.apache.solr.core.SolrCore execute
INFO: [collection1] webapp=/solr path=/admin/logging params={since=1353591348955&wt=json} status=0 QTime=0 
22-Nov-2012 08:36:25 org.apache.solr.handler.dataimport.JdbcDataSource closeConnection
SEVERE: Ignoring Error when closing connection
java.sql.SQLException: Unknown character set index for field '3687' received from server.
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:988)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:974)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
    at com.mysql.jdbc.ConnectionImpl.getCharsetNameForIndex(ConnectionImpl.java:2976)
    at com.mysql.jdbc.Field.<init>(Field.java:200)
    at com.mysql.jdbc.MysqlIO.unpackField(MysqlIO.java:758)
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:423)
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3096)
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2266)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2687)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2713)
    at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:4893)
    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4779)
    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4388)
    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1594)
    at org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:400)
    at org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:391)
    at org.apache.solr.handler.dataimport.DocBuilder.closeEntityProcessorWrappers(DocBuilder.java:291)
    at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:280)
    at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:382)
    at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:448)
    at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:429)
22-Nov-2012 08:36:25 org.apache.solr.common.SolrException log
SEVERE: Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
    at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:273)
    at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:382)
    at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:448)
    at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:429)
Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:413)
    at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:326)
    at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:234)
    ... 3 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:542)
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:411)
    ... 5 more
Caused by: java.lang.OutOfMemoryError: Java heap space

22-Nov-2012 08:36:25 org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: start rollback{flags=0,_version_=0}
22-Nov-2012 08:36:25 org.apache.solr.handler.dataimport.SolrWriter rollback
SEVERE: Exception while solr rollback.
java.lang.NullPointerException
    at org.apache.solr.update.DefaultSolrCoreState.rollbackIndexWriter(DefaultSolrCoreState.java:173)
    at org.apache.solr.update.DirectUpdateHandler2.rollbackWriter(DirectUpdateHandler2.java:150)
    at org.apache.solr.update.DirectUpdateHandler2.rollback(DirectUpdateHandler2.java:625)
    at org.apache.solr.update.processor.RunUpdateProcessor.processRollback(RunUpdateProcessorFactory.java:98)
    at org.apache.solr.update.processor.UpdateRequestProcessor.processRollback(UpdateRequestProcessor.java:72)
    at org.apache.solr.update.processor.LogUpdateProcessor.processRollback(LogUpdateProcessorFactory.java:170)
    at org.apache.solr.handler.dataimport.SolrWriter.rollback(SolrWriter.java:117)
    at org.apache.solr.handler.dataimport.DocBuilder.rollback(DocBuilder.java:319)
    at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:387)
    at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:448)
    at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:429)

What puzzles me is this line Unknown character set index for field '3687' received from server.. There is no field 3687 in any of the tables!

I'm basically setting up a new server and trying to migrate to this from the old one. I provisioned the new server using Puppet and transferred all the databases over to the MySQL database on the new server.

Server is an Ubuntu 12.04 running MySQL 5.5.28-0ubuntu0.12.04.2, Tomcat 7.0.26, Solr 4.0.0.2012.08.06.22.50.47. The older server was an Ubuntu 12.04 running MySQL 5.5.24

Any idea what went wrong with the full import? Thank you!

Answer 1

it was due to a lack of JVM memory and errors in the SQL query used in the full import