How to fix Multi-Master Replication Error in MariaDB

Question

I have a Multi-Master Ring Replication setup in MariaDB. 3 Servers.

One of my server's ran out of disk space and I eventually needed to restart the server. Now after doing that the two slave servers are reporting this error in the slave status.

Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Error: connecting slave requested to start from GTID 1-1-426253, which is not in the master's binlog'

I'm really confused on how to recover my slave from this error, could someone please tell me how I tell this slave server where to start from the correct GTID on it's master?

Thanks

Answer 1

I got it all working again. I simply found the masters log and position number by going to the master server and typing SHOW MASTER STATUS;

I then used that information on the slave and did this.

STOP SLAVE 'MDB1';

CHANGE MASTER "MDB1" TO master_host="xxx.xxx.xxx.xxx", master_port=3306, master_user="****", master_password="****", master_log_file="mariadb-bin.000394", master_log_pos=385;

START SLAVE 'MDB1';

Then checked it was all working ok, then I changed back to using GTID

STOP SLAVE 'MDB1';

CHANGE MASTER "MDB1" TO master_use_gtid=slave_pos;

START SLAVE 'MDB1'; 

After that it was all back and running again.

Answer 2

Moshe L, here is my masters binlog setup

server-id = 1
gtid_domain_id = 1
gtid_strict_mode = 1
report_host             = MDB1
auto_increment_increment = 3
auto_increment_offset   = 1
slave_parallel_threads = 12
replicate_ignore_db     = mysql
replicate_ignore_table  = MA4_Data.EOD_FileCache
log_bin                 = /var/log/mysql/mariadb-bin
log_bin_index           = /var/log/mysql/mariadb-bin.index
binlog_format           = mixed
#binlog_commit_wait_count = 12
#binlog_commit_wait_usec = 10000
#slave_compressed_protocol = 1
# not fab for performance, but safer
sync_binlog             = 1
expire_logs_days        = 10
max_binlog_size         = 100M
# slaves
relay_log               = /var/log/mysql/relay-bin
relay_log_index = /var/log/mysql/relay-bin.index
relay_log_info_file     = /var/log/mysql/relay-bin.info

Answer 3

This is another solution one could try.

stop slave; 
reset slave;
start slave