I've got a script that connects via SSH to the remote server, runs some commands to export the databases, archives them as a single file and then downloads via SCP to the local server, which drops the databases and then imports them.
The local system is basically a mirror image of the remote, production system except for the large datasets which do not need to be present for development.
#!/bin/bash
echo "Importing..."
Hostname=xxxxxx
Port=xxxxxx
Username=xxxxxx
Password=xxxxxx
Location=/sql/dump/$(date +"%Y-%m-%d")
ssh -p $Port $Username@$Hostname "mkdir -p $Location"
ssh -p $Port $Username@$Hostname <<'SSH'
Ignore=$(mysql --user=root --password=$Password -BNe "SELECT CONCAT( TABLE_SCHEMA , '.' , TABLE_NAME ) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'rec%' ORDER BY CONCAT( TABLE_SCHEMA , '.' , TABLE_NAME );" | awk '{ printf " --ignore-table=%s", $0 }')
mysql --user=root --password=$Password -e "SHOW DATABASES;" | grep -Ev "information_schema|mysql|performance_schema" | xargs mysqldump --force --user=root --password=$Password $Ignore --databases | gzip > $Location/full.sql.gz
SSH
mkdir -p $Location
scp -P$Port -c blowfish $Username@$Hostname:$Location/full.sql.gz $Location
ssh -p $Port $Username@$Hostname "rm -rf $Location" >/dev/null 2>&1
gunzip $Location/full.sql.gz
mysql --user=root --password=$Password -e "SHOW DATABASES;" | grep -Ev "information_schema|mysql|performance_schema" | awk '{print "DROP DATABASE " $1 "; SELECT SLEEP(0.1);"}' | mysql --user=root --password=$Password
mysql --user=root --password=$Password < $Location/full.sql
rm -rf $Location
echo "Done."
At the moment, everything works except this part:
Ignore=$(mysql --user=root --password=$Password -BNe "SELECT CONCAT( TABLE_SCHEMA , '.' , TABLE_NAME ) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'rec%' ORDER BY CONCAT( TABLE_SCHEMA , '.' , TABLE_NAME );" | awk '{ printf " --ignore-table=%s", $0 }')
mysql --user=root --password=$Password -e "SHOW DATABASES;" | grep -Ev "information_schema|mysql|performance_schema" | xargs mysqldump --force --user=root --password=$Password $Ignore --databases | gzip > $Location/full.sql.gz
What this is supposed to do is grab a list of all table names that start with "rec" and exclude them from the import (because they are huge and do not need to be stored locally). I feel that I am close but it is producing errorneous syntax in my mysqldump command.
Importing...
Pseudo-terminal will not be allocated because stdin is not a terminal.
Linux xxxxxx 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
zsh: permission denied: /full.sql.gz
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
Usage: mysqldump [OPTIONS] database [tables]
OR mysqldump [OPTIONS] --databases [OPTIONS] DB1 [DB2 DB3...]
OR mysqldump [OPTIONS] --all-databases [OPTIONS]
For more options, use mysqldump --help
scp: /sql/dump/2014-05-15/full.sql.gz: No such file or directory
gzip: /sql/dump/2014-05-15/full.sql.gz: No such file or directory
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'Database' at line 1
_private/import2.sh: line 23: /sql/dump/2014-05-15/full.sql: No such file or directory
Done.
I am looking for a way to exclude those tables and the system databases in my export, but perhaps more elegantly: getting rid of the heredoc syntax because it produces unwanted output in my STDOUT and also doesn't seem to accept local variables, I used it to get around some weird quote-escaping issues.
Assuming I do set the variables correctly, this is the output I get:
mysqldump: Got error: 1102: "Incorrect database name ' --ignore-table=0004_reset_manager.rec_alert --ignore-table=0005_reset_manager.rec_alert --ignore-ta'" when selecting the database
xargs: mysqldump: terminated by signal 7
UPDATE: It works! For those interested in using this on their server, here is the final script:
#!/bin/bash
echo "Importing..."
Hostname=xxxxxx
Port=xxxxxx
Username=xxxxxx
Password=xxxxxx
Directory=xxxxx
Filename=xxxxxx-$(date +"%Y-%m-%d")
Databases=$(ssh -p $Port $Username@$Hostname "mysql --user=root --password=$Password -BNe \"SHOW DATABASES;\" | grep -Ev \"information_schema|mysql|performance_schema\" | awk '{ printf \" %s\", \$0 }'")
Ignore=$(ssh -p $Port $Username@$Hostname "mysql --user=root --password=$Password -BNe \"SELECT CONCAT( TABLE_SCHEMA , '.' , TABLE_NAME ) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'rec%';\" | awk '{ printf \" --ignore-table=%s\", \$0 }'")
ssh -p $Port $Username@$Hostname "mysqldump --force --user=root --password=$Password $Ignore --databases $Databases | gzip > $Directory/$Filename.sql.gz"
scp -P$Port -c blowfish $Username@$Hostname:$Directory/$Filename.sql.gz $Directory
ssh -p $Port $Username@$Hostname "rm -f $Directory/$Filename.sql.gz" >/dev/null 2>&1
gunzip $Directory/$Filename.sql.gz
mysql --user=root --password=$Password -Ne "SHOW DATABASES;" | grep -Ev "information_schema|mysql|performance_schema" | awk '{ printf "DROP DATABASE %s;", $0 }' | mysql --user=root --password=$Password
mysql --user=root --password=$Password < $Directory/$Filename.sql
rm -f $Directory/$Filename.sql
echo "Done."