What is a LSN (log sequence number)? How does xtrabackup Incremental Backup work?

Question

I was surprised that xtrabackup supports incremental backups. xtrabackup copies the data files and understand the format of data files (can have a look inside)? What is a LSN for dummies? I found What is log sequence number ? How it is used in MySQL?.

Quote from Incremental Backup

An incremental backup copies each page whose LSN is newer than the previous incremental or full backup’s LSN. There are two algorithms in use to find the set of such pages to be copied. The first one, available with all the server types and versions, is to check the page LSN directly by reading all the data pages. The second one, available with Percona Server, is to enable the changed page tracking feature on the server, which will note the pages as they are being changed. This information will be then written out in a compact separate so-called bitmap file. The xtrabackup binary will use that file to read only the data pages it needs for the incremental backup, potentially saving many read requests. The latter algorithm is enabled by default if the xtrabackup binary finds the bitmap file. It is possible to specify xtrabackup --incremental-force-scan to read all the pages even if the bitmap data is available.

I still don't understand how this works. Do you use xtrabackup Incremental Backup? Do you recommend xtrabackup Incremental Backup?

Answer 1

LSN a.k.a. Log Sequence Number is a 8-bytes integer that acts as a version to REDO log entries(ib_logfile*). Every change to data makes an entry in the REDO log and gets a new LSN. The LSN is incremented with every change.

Every InnoDB index page (where user data are stored) has the LSN field in the header. It's an LSN that last modified the page.

So. If we know what the last LSN (let's call it lsn_full) was when we took a full backup, we know if a particular page changed since then. Because if the page's LSN (let's call it lsn_current) higher than lsn_full, the page has changed since the last backup.

Xtrabackup uses this idea to implement incremental backups. As a matter of fact you can implement both incremental and differential backups with Xtrabackup. Check out options --incremental-lsn and --incremental-basedir.

Whether use incremental backups or not depends on how confident you're about the whole process. It's easy to mess up if you don't fully understand what's going on during the backup and restoration process. (At TwinDB I have to deal with lots of cases where incremental backups were not usable on whatever reason).

I highly recommend verifying backups. The most reliable way to ensure a backup copy is usable is to restore the database from it. So I would suggest algorithm:

• grab the latest copy.
• restore a start MySQL instance on it.
• Check some table if it contains recent data. For example, if you store transactions in the database make sure there are some not older than a day or something.