Script to import databases from a remote server

Question

I've got a script that connects via SSH to the remote server, runs some commands to export the databases, archives them as a single file and then downloads via SCP to the local server, which drops the databases and then imports them.

The local system is basically a mirror image of the remote, production system except for the large datasets which do not need to be present for development.

#!/bin/bash

echo "Importing..."

Hostname=xxxxxx
Port=xxxxxx
Username=xxxxxx
Password=xxxxxx
Location=/sql/dump/$(date +"%Y-%m-%d")

ssh -p $Port $Username@$Hostname "mkdir -p $Location"

ssh -p $Port $Username@$Hostname <<'SSH'
Ignore=$(mysql --user=root --password=$Password -BNe "SELECT CONCAT( TABLE_SCHEMA , '.' , TABLE_NAME ) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'rec%' ORDER BY CONCAT( TABLE_SCHEMA , '.' , TABLE_NAME );" | awk '{ printf " --ignore-table=%s", $0 }')
mysql --user=root --password=$Password -e "SHOW DATABASES;" | grep -Ev "information_schema|mysql|performance_schema" | xargs mysqldump --force --user=root --password=$Password $Ignore --databases | gzip > $Location/full.sql.gz
SSH

mkdir -p $Location
scp -P$Port -c blowfish $Username@$Hostname:$Location/full.sql.gz $Location
ssh -p $Port $Username@$Hostname "rm -rf $Location" >/dev/null 2>&1
gunzip $Location/full.sql.gz
mysql --user=root --password=$Password -e "SHOW DATABASES;" | grep -Ev "information_schema|mysql|performance_schema" | awk '{print "DROP DATABASE " $1 "; SELECT SLEEP(0.1);"}' | mysql --user=root --password=$Password
mysql --user=root --password=$Password  < $Location/full.sql
rm -rf $Location

echo "Done."

At the moment, everything works except this part:

Ignore=$(mysql --user=root --password=$Password -BNe "SELECT CONCAT( TABLE_SCHEMA , '.' , TABLE_NAME ) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'rec%' ORDER BY CONCAT( TABLE_SCHEMA , '.' , TABLE_NAME );" | awk '{ printf " --ignore-table=%s", $0 }')
mysql --user=root --password=$Password -e "SHOW DATABASES;" | grep -Ev "information_schema|mysql|performance_schema" | xargs mysqldump --force --user=root --password=$Password $Ignore --databases | gzip > $Location/full.sql.gz

What this is supposed to do is grab a list of all table names that start with "rec" and exclude them from the import (because they are huge and do not need to be stored locally). I feel that I am close but it is producing errorneous syntax in my mysqldump command.

Importing...
Pseudo-terminal will not be allocated because stdin is not a terminal.
Linux xxxxxx 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
zsh: permission denied: /full.sql.gz
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
Usage: mysqldump [OPTIONS] database [tables]
OR     mysqldump [OPTIONS] --databases [OPTIONS] DB1 [DB2 DB3...]
OR     mysqldump [OPTIONS] --all-databases [OPTIONS]
For more options, use mysqldump --help
scp: /sql/dump/2014-05-15/full.sql.gz: No such file or directory
gzip: /sql/dump/2014-05-15/full.sql.gz: No such file or directory
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'Database' at line 1
_private/import2.sh: line 23: /sql/dump/2014-05-15/full.sql: No such file or directory
Done.

I am looking for a way to exclude those tables and the system databases in my export, but perhaps more elegantly: getting rid of the heredoc syntax because it produces unwanted output in my STDOUT and also doesn't seem to accept local variables, I used it to get around some weird quote-escaping issues.

Assuming I do set the variables correctly, this is the output I get:

mysqldump: Got error: 1102: "Incorrect database name ' --ignore-table=0004_reset_manager.rec_alert --ignore-table=0005_reset_manager.rec_alert --ignore-ta'" when selecting the database
xargs: mysqldump: terminated by signal 7

UPDATE: It works! For those interested in using this on their server, here is the final script:

#!/bin/bash

echo "Importing..."

Hostname=xxxxxx
Port=xxxxxx
Username=xxxxxx
Password=xxxxxx
Directory=xxxxx
Filename=xxxxxx-$(date +"%Y-%m-%d")

Databases=$(ssh -p $Port $Username@$Hostname "mysql --user=root --password=$Password -BNe \"SHOW DATABASES;\" | grep -Ev \"information_schema|mysql|performance_schema\" | awk '{ printf \" %s\", \$0 }'")
Ignore=$(ssh -p $Port $Username@$Hostname "mysql --user=root --password=$Password -BNe \"SELECT CONCAT( TABLE_SCHEMA , '.' , TABLE_NAME ) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'rec%';\" | awk '{ printf \" --ignore-table=%s\", \$0 }'")

ssh -p $Port $Username@$Hostname "mysqldump --force --user=root --password=$Password $Ignore --databases $Databases | gzip > $Directory/$Filename.sql.gz"
scp -P$Port -c blowfish $Username@$Hostname:$Directory/$Filename.sql.gz $Directory
ssh -p $Port $Username@$Hostname "rm -f $Directory/$Filename.sql.gz" >/dev/null 2>&1
gunzip $Directory/$Filename.sql.gz
mysql --user=root --password=$Password -Ne "SHOW DATABASES;" | grep -Ev "information_schema|mysql|performance_schema" | awk '{ printf "DROP DATABASE %s;", $0 }' | mysql --user=root --password=$Password
mysql --user=root --password=$Password < $Directory/$Filename.sql
rm -f $Directory/$Filename.sql

echo "Done."

Answer 1

It's not working because it's not interpolating the variables. remove quotes around SSH and it should work. But mind you that you will have to escape $ and possibly other characters as well.

Example:

$cat test.sh 
a=test

cat << 'SSH'
$a
EOF

cat << SSH
$a 
EOF
$ bash test.sh 
$a
test

Try this:

ssh -p $Port $Username@$Hostname << SSH
Ignore=$(mysql --user=root --password=$Password -BNe "SELECT CONCAT( TABLE_SCHEMA , '.' ,  TABLE_NAME ) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'rec%' ORDER BY CONCAT( TABLE_SCHEMA , '.' , TABLE_NAME );" | awk '{ printf " --ignore-table=%s", \$0 }')
mysql --user=root --password=$Password -e "SHOW DATABASES;" | grep -Ev "information_schema|mysql|performance_schema" | xargs mysqldump --force --user=root --password=$Password \$Ignore --databases | gzip > $Location/full.sql.gz

SSH

I can't test but it should be fine.