upload csv file containing million values to a particular column in mysql

Question

Database: MySQL

CSV file having around million records like the following:

234546,345674,464368,563236,684269,707853,...

Data should get added into db in following manner:

id|number|status| ...
-----------------------
 1|234546| ...  | ...
 2|345674| ...  | ...
 3|464368| ...  | ...
 4|563236| ...  | ...
 5|684269| ...  | ...
 6|707853| ...  | ...

I've found some ways of uploading a csv file when the csv file contains values for each column. But here in my case, I have to insert values to a particular column. What is the efficient way of doing this in java?

Answer 1

The fastest way to load data from file is LOAD DATA INFILE. Try it this way

LOAD DATA LOCAL INFILE '/path/to/your/file.csv'
IGNORE -- use this if you have duplicate numbers in the file and UNIQUE constraint defined on `number` column
INTO TABLE table1
FIELDS TERMINATED BY ','
LINES TERMINATED BY ','
(number)

Let's try it

mysql> CREATE TABLE Table1
    -> (
    ->   `id` int not null auto_increment primary key, 
    ->   `number` int, 
    ->   `status` varchar(32)
    -> );
Query OK, 0 rows affected (0.13 sec)

mysql> 
mysql> LOAD DATA LOCAL INFILE '/tmp/test.csv'
    -> INTO TABLE table1
    -> FIELDS TERMINATED BY ','
    -> LINES TERMINATED BY ','
    -> (number);
Query OK, 6 rows affected (0.00 sec)
Records: 6  Deleted: 0  Skipped: 0  Warnings: 0

mysql> select * from table1;
+----+--------+--------+
| id | number | status |
+----+--------+--------+
|  1 | 234546 | NULL   |
|  2 | 345674 | NULL   |
|  3 | 464368 | NULL   |
|  4 | 563236 | NULL   |
|  5 | 684269 | NULL   |
|  6 | 707853 | NULL   |
+----+--------+--------+
6 rows in set (0.00 sec)

---

UPDATE: If you need to do some basic transformations or validations you can use SET clause along with user variables

LOAD DATA LOCAL INFILE '/path/to/your/file.csv'
IGNORE INTO TABLE table1
FIELDS TERMINATED BY ','
LINES TERMINATED BY ','
(@number)
SET number = CONCAT('prefix', @number)

Answer 2

1. Make id field AUTO_INCREMENT.
2. Set default values for other fields.
3. Use LOAD DATA INFILE to insert only one field - number, and set ',' as a line separator.

Answer 3

The best way is to use the LOAD DATA INFILE technology in MySQL.

for eg:

String path = "o.csv";
                String esquel = " LOAD DATA LOCAL INFILE " + path +
                            " INTO TABLE mytable" +
                            " LINES TERMINATED BY \'\\n\'";

statement.executeUpdate(esquel);

Rewrite the query correctly for your requirement. This do a direct load instead of using the JAVA JDBC API for inserting.

Answer 4

I suggest using bash shell script to handle this problem directly, instead of using Java. We may follow the steps below:

First generate SQL statements to insert:

tr ',' '\n' < text.csv | grep -v "^[ \t]*$" | awk '{print "INSERT INTO YOURTABLE (id, number) VALUES ("NR ", " $1 ")"}' > insert.sql

• tr ',' '\n' < text.csv : Replace ',' with the newline '\n';
• grep -v "^[ \t]*$" : Remove empty lines;
• awk ... > insert.sql : Add the SQL keywords into it.

The result may look like this:

INSERT INTO YOURTABLE (id, number) VALUES (1, 234546)

INSERT INTO YOURTABLE (id, number) VALUES (2, 345674)

INSERT INTO YOURTABLE (id, number) VALUES (3, 464368)

INSERT INTO YOURTABLE (id, number) VALUES (4, 563236)

Then execute these SQL insertions:

mysql -uUSERNAME -pPASSWORD -hHOST < insert.sql