Xtrabackup corrupt incremental backup

Question

I have 3 nodes PXC 5.6 cluster. Xtrabackup 2.3.10. Centos 7.6

Percona-XtraDB-Cluster-56-5.6.41-28.28.1.el7.x86_64

I am using only single Node-1 for both read and write.

And I use to take backup from another Node-2 which is not used for read or write.

Yesterday, my Node-1 went down and read/write switched to Node-2. Then we continued to take incremental backup from another unused Node-3.

On my backup machine, I decided to extract and prepare backup. My backup preparation log shows everything fine until those backups that were taken from Node-2. All the incremental backups from Node-3 had errors.

Why does it seems like incremental backup should always be from the same node where the full backup was taken? Percona XtraDB Cluster is master-master and so all nodes must be same. Then why did this happen?

I have attached my logs here:

https://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/53285-xtrabackup-corrupts-incremental-backup-when-node-changes

Answer 1

LSN (Log Sequence Number), which incremental backup is based on, is specific to an instance. That's why knowing full copy LSN means nothing on other node.