“No valid checkpoint found” when restoring xtrabackup

Question

I have a cluster of MariaDB 10.1 servers, master/master with one node acting as arbitrator and incremental backups with xtrabackup.

This is how I've pieced it together from the docs.

Master01 does a base backup once using this command:

innobackupex --defaults-file="/etc/xtrabackup_client/backup.cnf" \
  --socket="..." --extra-lsndir="/etc/xtrabackup_client/" \
  --stream=xbstream /tmp | \
  ssh "$backup_username@$backup_server"  "cat - | xbstream -x -C /var/backups/xtrabackup/"

After that I do incremental backups every day with this command:

innobackupex --defaults-file="$mysql_defaults_file" \
  --socket="..." --extra-lsndir="/etc/xtrabackup_client" --stream=xbstream \
  --incremental --incremental-lsn="$to_lsn" /tmp | \
  ssh "$backup_username@$backup_server" "cat - | xbstream -x -C /var/backups/xtrabackup_incremental/"

/etc/xtrabackup_client/backup.cnf looks like this.

[client]
user=backup
password=secret

This is done in scripts, hence the variables here and there. (I've also removed some to hopefully clarify.)

This seems to work, I get a base backup and I get incremental backups created.

The problem happens when I try to do a restore test.

For restoring I follow this procedure on the arbitrator-node.

First I run a script that loops through the base backup and all the incremental backups in rising chronological order and verifies the LSN.

I've pasted the script in gist here.

If all looks good I run another script to merge the incremental backups, gist here.

This results in one base backup that by size looks the same as the live copy on the master servers.

I then want to test it so I do innobackupex --copy-back /var/backups/xtrabackup which copies the base backup into /var/db/mysql which is the datadir configured in /etc/mysql.

That server does not have a mysqld binary installed so I rsync the whole dir over to a restore server with the same MariaDB 10.1 version as the master servers.

Then I try to run it as root with this command:

sudo -u mysql /usr/sbin/mysqld --basedir=/usr \
  --datadir=/var/db/mysql --plugin-dir=/usr/lib/mysql/plugin \
  --user=mysql --pid-file=/var/run/mysqld/mysqld.pid \
  --socket=/var/run/mysqld/mysqld.sock --port=3306

But I get this output:

2017-01-03 18:55:41 140551642929088 [Note] /usr/sbin/mysqld (mysqld 10.1.20-MariaDB-1~trusty) starting as process 24455 ...
2017-01-03 18:55:41 140551642929088 [Note] Using unique option prefix 'myisam_recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
2017-01-03 18:55:41 140551642929088 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-01-03 18:55:41 140551642929088 [Note] InnoDB: The InnoDB memory heap is disabled
2017-01-03 18:55:41 140551642929088 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-01-03 18:55:41 140551642929088 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-01-03 18:55:41 140551642929088 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-01-03 18:55:41 140551642929088 [Note] InnoDB: Using Linux native AIO
2017-01-03 18:55:41 140551642929088 [Note] InnoDB: Using SSE crc32 instructions
2017-01-03 18:55:41 140551642929088 [Note] InnoDB: Initializing buffer pool, size = 256.0M
2017-01-03 18:55:41 140551642929088 [Note] InnoDB: Completed initialization of buffer pool
2017-01-03 18:55:41 140551642929088 [Note] InnoDB: Highest supported file format is Barracuda.
InnoDB: No valid checkpoint found.
InnoDB: If you are attempting downgrade from MySQL 5.7.9 or later,
InnoDB: please refer to http://dev.mysql.com/doc/refman/5.6/en/upgrading-downgrading.html
InnoDB: If this error appears when you are creating an InnoDB database,
InnoDB: the problem may be that during an earlier attempt you managed
InnoDB: to create the InnoDB data files, but log file creation failed.
InnoDB: If that is the case, please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/error-creating-innodb.html
2017-01-03 18:55:41 140551642929088 [ERROR] Plugin 'InnoDB' init function returned error.
2017-01-03 18:55:41 140551642929088 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2017-01-03 18:55:41 140551642929088 [Note] Plugin 'FEEDBACK' is disabled.
2017-01-03 18:55:41 140551642929088 [ERROR] Unknown/unsupported storage engine: InnoDB
2017-01-03 18:55:41 140551642929088 [ERROR] Aborting

Which makes little sense to me so I'm here asking for help.

Answer 1

@jcolebrand and some more googling helped steer me in the right direction on this one.

InnoDB engine is failing to start. So Show engines showed InnoDB as not supported if I started with myisam as default engine.

This github issue hinted that a few old logfiles had to be removed.

Specifically I removed the following files but the number of ib_logfile files depends on your system.

• aria_log_control
• ib_logfile0
• ib_logfile1

Move them to another dir and try starting the service again. This worked for me.