Derby SQL Script Bulk Import

Question

my goal is to import data into a derby database. The data was extracted from a MySQL instance in form of a SQL dump and I took care with a script that I only use the insert-statements and also that the escaping of the MySQL specific syntax is transformed to Derby correctly. To Test, I use the derby-maven-plugin:

export MAVEN_OPTS=-Xmx2048m; mvn derby:run

In the first step, I create the schema in the derby instance (using DDLUtils from MySQL, this works fine). Second, I try to import the data. I use ij on the command line with the following (shortend) script:

CONNECT 'jdbc:derby://localhost:1527/foodmart';
SET SCHEMA APP;
autocommit off;
INSERT INTO "ACCOUNT" VALUES (1000,NULL,'Assets','Asset','~',NULL),                            (2000,NULL,'Liabilities','Liability','~',NULL),(3000,5000,'Net Sales','Income','+',NULL),(3100,3000,'Gross Sales','Income','+','LookUpCube(\"[Sales]\",\"(Measures.[Store Sales],\"+time.currentmember.UniqueName+\",\"+ Store.currentmember.UniqueName+\")\")'),(3200,3000,'Cost of Goods Sold','Income','-',NULL),(4000,5000,'Total Expense','Expense','-',NULL),(4100,4000,'General & Administration','Expense','+',NULL),(4200,4000,'Information Systems','Expense','+',NULL),(4300,4000,'Marketing','Expense','+',NULL),(4400,4000,'Lease','Expense','+',NULL),(5000,NULL,'Net Income','Income','+',NULL);
...
commit;

As you can see, for every row of a table there is a bracket containing the sample data. There are of course several more insert statments for other tables. After finishing some inserts correctly, the bulk importing process chokes at a really big dataset (> 1000 rows) due to following exception (from the derby-logs):

java.lang.StackOverflowError
at org.apache.derby.impl.sql.compile.SetOperatorNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.UnionNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.TableOperatorNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.SetOperatorNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.UnionNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.TableOperatorNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.SetOperatorNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.UnionNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.TableOperatorNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.SetOperatorNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.UnionNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.TableOperatorNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.SetOperatorNode.bindExpressions(Unknown Source)
at org.apache.derby.impl.sql.compile.UnionNode.bindExpressions(Unknown Source)
... and many lines more repeatingly because of the recursive nature...
Cleanup action completed 

ij prints on the command line:

FEHLER XJ001: DERBY SQL error: SQLCODE: -1, SQLSTATE: XJ001, SQLERRMC:  java.lang.StackOverflowError^T^TXJ001.U

Is this meaning, that derby is not capable using SQL-bulk-import scenarios? Should I instead switching to an csv based import? I assume from the stacktrace method calls that derby is not able to create the query plan. Could it be possible, that for every 'data bracket' of an insert statement it will create internly an UNION statement, adding a big amount of overhead to each INSERT query. So should I try to split my long INSERT-statements in many concise statements? I don't have the time to look at the derby sources, so please help me!

Answer 1

I just used many Import statements and it worked.

It seems that for every data bracket of an INSERT statement Derby generates a method call on the stack, leeding for many brackets to a stackoverflow due to many recursive method calls.

However You just need to transform an INSERT statement from

Insert INTO VALUES (1,2,3), ..., (4,5,6);

To multiple statements:

Insert INTO VALUES (1,2,3); ... Insert INTO VALUES (4,5,6);