How to find error rows in bulk insert?

Question

MySQL 5.6: I'm experimenting with copying data from one table to another in different ways. The source table is on a remote server and has about 500,000 rows - I use the federated engine to connect. I first tried this:

mysql > create table tgt as select * from src;

This is very fast, takes only a few seconds, but it gives warnings:

...
| Warning | 1299 | Invalid TIMESTAMP value in column 'created_timestamp' at row 265975 |
| Warning | 1299 | Invalid TIMESTAMP value in column 'created_timestamp' at row 265976 |
...
64 rows in set (0.00 sec)

I tried instead to do it with a stored procedure, opening a cursor, fetching rows and inserting them, but it takes forever; I canceled after 10 min.

So, is there a way to locate the rows that cause the problem? I tried select ... limit #first_row,#last_row;, but it doesn't appear to work, and I'm not sure if it is entirely reliable any way.

Answer 1

To locate the rows you want to see, you must do two things:

• change the way you create the table
• change the way you load data into the table

Why Change the Way You Create the Table ???

When you did this

create table tgt as select * from src;

You create the tgt table without any indexes. You can verify this by running

show create table src \G
show create table tgt \G

and you will see src with its indexes and tgt without them.

Consequently, with no indexes around, the table load would be the fastest. If you at least had a primary key. you could navigate the table by some numeric id column.

So, to create table with the indexes in place, do this:

create table tgt like src;

You could then run

show create table src \G
show create table tgt \G

and see that they are the same in structure.

Why Change the Way You Load Data into the Table ???

You could just start loading like this

insert into tgt select * from src;

When the load is done, you could run

SELECT * FROM tgt WHERE id = 265975;
SELECT * FROM tgt WHERE id = 265976;

You could get an glimpse at when the row looks like but the created_timestamp may not be populated. If it is populated correctly, then the problem is essentially solved.

Another Way to See Which Row Without Loading Data

You could isolate which rows have an issue with the created_timestamp by doing something else

# mysqldump --skip-extended-insert mydb src > dumpfile.txt
# head -265976 dumpfile.txt | tail -2

This will show you the rows that would have been inserted.

This is all the help I can suggest since I do not know what the table structures are.