which one is fast load data infile or normal insertion?

Question

I have a table with 40 million records, and I want to copy this 40M into 80M like this-

INSERT INTO cdr (id,dataPacketDownLink, dataPacketUpLink,dataPlanEndTime,dataPlanStartTime,dataVolumeDownLink,dataVolumeUpLink,
dataplan,dataplanType, createdOn, deviceName, duration, effectiveDuration, HOUR, eventDate, msisdn, QUARTER, validDays, dataLeft,completedOn,evedate)SELECT null,dataPacketDownLink, dataPacketUpLink,dataPlanEndTime,dataPlanStartTime,dataVolumeDownLink,dataVolumeUpLink,
dataplan,dataplanType, createdOn, deviceName, duration, effectiveDuration, HOUR, eventDate, msisdn, QUARTER, validDays, dataLeft,completedOn,eve_date FROM cdr;

is it correct to use this one or I should use SELECT INTO OUTFILE and LOAD DATA INFILE for fast insertion? Or is there any better procedure than above two?

Thank you.

Answer 1

To avoid the big TEMPORARY TABLE created implicitely in this operation (inserting into the table being selected), you can try this option:

CREATE TABLE cdr2 AS
SELECT dataPacketDownLink, dataPacketUpLink,dataPlanEndTime,dataPlanStartTime,dataVolumeDownLink,dataVolumeUpLink,
dataplan,dataplanType, createdOn, deviceName, duration, effectiveDuration, HOUR, eventDate, msisdn, QUARTER, validDays, dataLeft,completedOn,evedate 
FROM cdr
UNION ALL
SELECT dataPacketDownLink, dataPacketUpLink,dataPlanEndTime,dataPlanStartTime,dataVolumeDownLink,dataVolumeUpLink,
dataplan,dataplanType, createdOn, deviceName, duration, effectiveDuration, HOUR, eventDate, msisdn, QUARTER, validDays, dataLeft,completedOn,evedate 
FROM cdr;

ALTER TABLE cdr2 ADD COLUMN (id INT(10) NOT NULL UNIQUE AUTO_INCREMENT)

once you checked that everything is ok in cdr2, you may:

• DROP TABLE cdr
• RENAME TABLE cdr2 TO cdr

Don't forget to add the missing indexes to the new table.

Answer 2

You current solution is fairly good, but if your table has a lot of indexes, that might slow things down. Indexes are good for searching, but slow down insertion.

You could try dropping all (or most) of the indexes first, copy all data, and then re-create them. Alternatively, you could construct a new table without indexes, copy all the data there, create indexes on the new table and then drop the old table and rename the new table.

Nevertheless, I've seen modifications on tables this size taking this long before. It's unfortunate, but MySQL isn't very fast at these things.