Edit very large sql dump/text file (on linux)

Question

I have to import a large mysql dump (up to 10G). However the sql dump already predefined with a database structure with index definition. I want to speed up the db insert by removing the index and table definition.

That means I have to remove/edit the first few lines of a 10G text file. What is the most efficient way to do this on linux?

Programs that require loading the entire file into RAM will be an overkill to me.

Answer 1

Rather than removing the first few lines, try editing them to be whitespace.

The hexedit program can do this-- it reads files in chunks, so opening a 10GB file is no different from opening a 100KB file to it.

$ hexedit largefile.sql.dump
tab (switch to ASCII side)
space (repeat as needed until your header is gone)
F2 (save)/Ctrl-X (save and exit)/Ctrl-C (exit without saving)

Answer 2

joe is an editor that works well with large files. I just used it to edit a ~5G SQL dump file. It took about a minute to open the file and a few minutes to save it, with very little use of swap (on a system with 4G RAM).

Answer 3

sed 's/OLD_TEXT/NEW_TEXT/g' < oldfile > newfile

or

cat file | sed 's/OLD_TEXT/NEW_TEXT/g' > newfile

Answer 4

Perl can read the file line by line:

perl -pi.bak -e 's/^create index/--create index/'