Infobright/MySQL load data infile deadlock

Question

I am experiencing a deadlock when using multiple concurrent LOAD DATA INFILE comamnds. Here is the background:

I have developed a Java-based ETL tool which loads flat files into Infobright databases (Infobright is a columnar database based on MySQL). At the end of the processing, the load is executed using a LOAD DATA INFILE command. This process can handle loading into multiple tables and each load is done in parallel using a separate connection (for performance reasons).

I also have written a bash script which runs this tool in parallel for multiple files (to save time on the processing part, since the LOAD commands will be serialized anyway by the DB server). This means i can have multiple LOAD DATA commands for the same target table (from different processes).

What i was expecting was that the LOAD DATA command would be executed in a serial fashion, but would eventually finish. However, i am experiencing a deadlock. If i execute a "show processlist" i can see that all my LOAD DATA commands are in the "System locked" state.

Has anyone experienced such a problem before and found a solution?

Details: I'm using Infobright Enterprise Edition v3.5.2. This is based on
MySQL 5.1.40. My ETL application uses java 1.5.0_08 and Mysql
Connector Java v5.1.12. OS: Centos 5.6 64 bit (Linux 2.6.18-238.12.1.el5)

Answer 1

Please make sure you have enough memory on your system to run the ETL process and script. From external investigation with you, it was determined you were running on a machine with only 1GB RAM ([1002M/498M, 1/0.00]) which is very low. Try testing on a regular desktop class machine (>4G of RAM), as well as on a production class machine (>32G of RAM).