MySQL INSERT INTO SELECT vs BULK INSERTs performance
-
04-03-2021 - |
سؤال
Currently, I am trying to copy data from TABLE1
to TABLE2
.
In terms of insertions performance, would it be the same or faster if I would to do
BULK INSERT
manually (i.e BULK insert every 10K records into TABLE2 viaINSERT INTO TABLE2 VALUES (1,2), (5,5), ...
), versusINSERT INTO TABLE2 SELECT * FROM TABLE1
المحلول
You have to go with the BULK INSERT.
WHY NOT INSERT INTO TABLE2 SELECT * FROM TABLE1
???
Running INSERT INTO TABLE2 SELECT * FROM TABLE1
requires a single transaction.
Imaging how populated an undo log will be to perform a single rollback.
If that transaction fails and rolls back, you create lots of table fragmentation.
Why BULK INSERT
manually ???
This takes a lot of pressure off the InnoDB Storage Engine for holding large undo information.
EXAMPLE : mysqldump
Have you ever noticed when reloading a mysqldump, hundreds or thousands of rows at a time are being inserted ? If you grep a mysqldump like this:
grep "^INSERT" dump.sql
You will see many lines with INSERTs. Each INSERT is an extended insert by default. That allows 100's of rows to be inserted per INSERT command. So, the principle you already suggested of BULK INSERT 10K rows at a time is perfectly acceptable.
نصائح أخرى
Do it in chunks. That is, walk through the source, copying 1K rows from the source to the destination. And COMMIT
. Do the walking based on the PRIMARY KEY
for maximal read efficiency.
I cover a lot of the issues here: http://mysql.rjweb.org/doc.php/deletebig
(It discusses DELETEs
, but the underlying principles work for SELECT
.)