Mariadb galera cluster nodes won't stay joined
https://dba.stackexchange.com/questions/110984
italiano
english
français
española
中国
日本の
العربية
Deutsch
한국어
Português
RussianFrage
I have a fresh install on Mariadb 10.0.20 on three servers. I'm using the same my.cnf from another running cluster, just changed the cluster name and the IPs. I bootstrap the first node, and I get a cluster size of 1. When starting the next node, the cluster size goes to 2 for a bit, syslog shows data being transfered, but then it says that there are zero mysql processes running, and shuts down.
my.cnf file on first node:
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /run/mysqld
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking
skip_name_resolve
bind-address = 0.0.0.0
max_connections = 1000
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 16M
thread_cache_size = 1280
sort_buffer_size = 4M
bulk_insert_buffer_size = 16M
tmp_table_size = 32M
max_heap_table_size = 32M
myisam_recover = BACKUP
key_buffer_size = 18M
#open-files-limit = 2000
table_open_cache = 400
myisam_sort_buffer_size = 52M
concurrent_insert = 2
read_buffer_size = 2M
read_rnd_buffer_size = 1M
query_cache_limit = 128K
query_cache_size = 0
/etc/mysql/conf.d/mysqld_safe_syslog.cnf.
log_warnings = 2
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
log_slow_verbosity = query_plan
log_bin = /var/log/mysql/mariadb-bin
log_bin_index = /var/log/mysql/mariadb-bin.index
expire_logs_days = 10
max_binlog_size = 100M
binlog_format = ROW
default_storage_engine = InnoDB
innodb_log_file_size = 512M
innodb_buffer_pool_size = 10G
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
innodb_thread_concurrency = 32
innodb_io_capacity = 2000
innodb_doublewrite = 0
innodb_use_fallocate = 1
innodb_buffer_pool_instances = 10
innodb_write_io_threads = 32
innodb_read_io_threads = 32
innodb_file_format = Barracuda
innodb_lru_scan_depth = 2000
innodb_log_group_home_dir = /mysql_logdir
innodb_data_home_dir = /mysql_datadir
innodb_autoinc_lock_mode = 2
innodb_flush_log_at_trx_commit = 0
wsrep_provider = /usr/lib/libgalera_smm.so
wsrep_provider_options = "gcache.size=300M; gcache.page_size=1G"
wsrep_cluster_name = "device_cluster"
wsrep_cluster_address = "gcomm://10.150.28.19, 10.150.28.20"
wsrep_sst_method = rsync
wsrep_node_address = "10.150.28.18"
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
syslog from 2nd node:
20:47:29 localhost mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
20:47:29 localhost mysqld_safe: WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.20E6uy' --pid-file='/var/lib/mysql/avc2-wus1-db05-recover.pid'
20:47:29 localhost mysqld: 150813 20:47:29 [Note] /usr/sbin/mysqld (mysqld 10.0.20-MariaDB-1~trusty-wsrep-log) starting as process 5443 ...
20:47:34 localhost mysqld_safe: WSREP: Recovered position 00000000-0000-0000-0000-000000000000:2
20:47:34 localhost mysqld: 150813 20:47:34 [Note] /usr/sbin/mysqld (mysqld 10.0.20-MariaDB-1~trusty-wsrep-log) starting as process 5558 ...
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: Read nil XID from storage engines, skipping position init
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: wsrep_load(): Galera 3.9(rXXXX) by Codership Oy <info@codership.com> loaded successfully.
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: CRC-32C: using hardware acceleration.
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:2
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: Passing config to GCS: base_host = 10.150.28.19; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto
_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_
install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /
var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 1G; gcache.size = 300M; gcs.fc_debug = 0; gcs.fc
_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_l
imit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npv
20:47:34 localhost mysqld: o = false; pc.recovery
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: Service thread queue flushed.
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: Assign initial position for certification: 2, protocol version: -1
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: wsrep_sst_grab()
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: Start replication
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:2
20:47:34 localhost mysqld: 150813 20:47:34 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_init_history(gcs_group_t*, gcs_seqno_t, const gu_uuid_t*)():80: Non-negativ
e state seqno requires non-nil history UUID.
20:47:34 localhost mysqld: 150813 20:47:34 [ERROR] WSREP: gcs init failed:Invalid argument
20:47:34 localhost mysqld: 150813 20:47:34 [ERROR] WSREP: wsrep::connect() failed: 7
20:47:34 localhost mysqld: 150813 20:47:34 [ERROR] Aborting
20:47:34 localhost mysqld:
20:47:34 localhost mysqld: 150813 20:47:34 [Note] WSREP: Service disconnected.
20:47:35 localhost mysqld: 150813 20:47:35 [Note] WSREP: Some threads may fail to exit.
20:47:35 localhost mysqld: 150813 20:47:35 [Note] /usr/sbin/mysqld: Shutdown complete
20:47:35 localhost mysqld:
20:47:35 localhost mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
20:47:43 localhost mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
20:47:43 localhost mysqld_safe: WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.zgu0Hd' --pid-file='/var/lib/mysql/avc2-wus1-db05-recove
r.pid'
20:47:43 localhost mysqld: 150813 20:47:43 [Note] /usr/sbin/mysqld (mysqld 10.0.20-MariaDB-1~trusty-wsrep-log) starting as process 6001 ...
20:47:48 localhost mysqld_safe: WSREP: Recovered position 00000000-0000-0000-0000-000000000000:2
20:47:48 localhost mysqld: 150813 20:47:48 [Note] /usr/sbin/mysqld (mysqld 10.0.20-MariaDB-1~trusty-wsrep-log) starting as process 6106 ...
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: Read nil XID from storage engines, skipping position init
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: wsrep_load(): Galera 3.9(rXXXX) by Codership Oy <info@codership.com> loaded successfully.
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: CRC-32C: using hardware acceleration.
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: Passing config to GCS: base_host = 10.150.28.19; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto
_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_
install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /
var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 1G; gcache.size = 300M; gcs.fc_debug = 0; gcs.fc
_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_l
imit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.np
v
20:47:48 localhost mysqld: o = false; pc.recovery
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: Service thread queue flushed.
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: wsrep_sst_grab()
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: Start replication
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: protonet asio version 0
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: Using CRC-32C for message checksums.
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: backend: asio
20:47:48 localhost mysqld: 150813 20:47:48 [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory)
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: restore pc from disk failed
20:47:48 localhost mysqld: 150813 20:47:48 [Note] WSREP: GMCast version 0
20:48:03 localhost mysqld: 150813 20:48:03 [Warning] WSREP: Failed to resolve tcp:// 10.150.28.20:4567
20:48:03 localhost mysqld: 150813 20:48:03 [Note] WSREP: (8e8c1fdb, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
20:48:03 localhost mysqld: 150813 20:48:03 [Note] WSREP: (8e8c1fdb, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
20:48:03 localhost mysqld: 150813 20:48:03 [Note] WSREP: EVS version 0
20:48:03 localhost mysqld: 150813 20:48:03 [Note] WSREP: gcomm: connecting to group 'device_cluster', peer '10.150.28.18:, 10.150.28.20:'
20:48:03 localhost mysqld: 150813 20:48:03 [Note] WSREP: (8e8c1fdb, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
20:48:03 localhost mysqld: 150813 20:48:03 [Note] WSREP: declaring 620d451d at tcp://10.150.28.18:4567 stable
20:48:03 localhost mysqld: 150813 20:48:03 [Note] WSREP: Node 620d451d state prim
20:48:03 localhost mysqld: 150813 20:48:03 [Note] WSREP: view(view_id(PRIM,620d451d,4) memb {
20:48:03 localhost mysqld: #011620d451d,0
20:48:03 localhost mysqld: #0118e8c1fdb,0
20:48:03 localhost mysqld: } joined {
20:48:03 localhost mysqld: } left {
20:48:03 localhost mysqld: } partitioned {
20:48:03 localhost mysqld: })
20:48:03 localhost mysqld: 150813 20:48:03 [Note] WSREP: save pc into disk
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: gcomm: connected
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Opened channel 'device_cluster'
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Waiting for SST to complete.
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: STATE EXCHANGE: sent state msg: 97aed7ec-41fc-11e5-8545-635104629780
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: STATE EXCHANGE: got state msg: 97aed7ec-41fc-11e5-8545-635104629780 from 0 (avc2-wus1-db04)
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: STATE EXCHANGE: got state msg: 97aed7ec-41fc-11e5-8545-635104629780 from 1 (avc2-wus1-db05)
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Quorum results:
20:48:04 localhost mysqld: #011version = 3,
20:48:04 localhost mysqld: #011component = PRIMARY,
20:48:04 localhost mysqld: #011conf_id = 3,
20:48:04 localhost mysqld: #011members = 1/2 (joined/total),
20:48:04 localhost mysqld: #011act_id = 2,
20:48:04 localhost mysqld: #011last_appl. = -1,
20:48:04 localhost mysqld: #011protocols = 0/7/3 (gcs/repl/appl),
20:48:04 localhost mysqld: #011group UUID = 76a27c1d-41f1-11e5-846a-73bafbb95904
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Flow-control interval: [23, 23]
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 2)
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: State transfer required:
20:48:04 localhost mysqld: #011Group state: 76a27c1d-41f1-11e5-846a-73bafbb95904:2
20:48:04 localhost mysqld: #011Local state: 00000000-0000-0000-0000-000000000000:-1
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: New cluster view: global state: 76a27c1d-41f1-11e5-846a-73bafbb95904:2, view# 4: Primary, number of nodes: 2, my index: 1, protocol version 3
20:48:04 localhost mysqld: 150813 20:48:04 [Warning] WSREP: Gap in state sequence. Need state transfer.
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.150.28.19' --auth '' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '6106' --binlog '/var/log/mysql/mariadb-bin' '
20:48:04 localhost rsyncd[6293]: rsyncd version 3.1.0 starting, listening on port 4444
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Prepared SST request: rsync|10.150.28.19:4444/rsync_sst
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: REPL Protocols: 7 (3, 2)
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Service thread queue flushed.
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Assign initial position for certification: 2, protocol version: 3
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Service thread queue flushed.
20:48:04 localhost mysqld: 150813 20:48:04 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (76a27c1d-41f1-11e5-846a-73bafbb95904): 1 (Operation not permitted)
20:48:04 localhost mysqld: #011 at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Member 1.0 (avc2-wus1-db05) requested state transfer from '*any*'. Selected 0.0 (avc2-wus1-db04)(SYNCED) as donor.
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 2)
20:48:04 localhost mysqld: 150813 20:48:04 [Note] WSREP: Requesting state transfer: success, donor: 0
20:48:04 localhost rsyncd[6308]: connect from bb-db-device-01 (10.150.28.18)
20:48:04 localhost rsyncd[6308]: rsync to rsync_sst/ from bb-db-device-01 (10.150.28.18)
20:48:04 localhost rsyncd[6308]: receiving file list
20:48:04 localhost rsyncd[6310]: connect from bb-db-device-01 (10.150.28.18)
20:48:04 localhost rsyncd[6308]: sent 63 bytes received 12,596,622 bytes total size 12,593,152
20:48:05 localhost rsyncd[6310]: rsync to rsync_sst-log_dir/ from bb-db-device-01 (10.150.28.18)
20:48:05 localhost rsyncd[6310]: receiving file list
20:48:06 localhost mysqld: 150813 20:48:06 [Note] WSREP: (8e8c1fdb, 'tcp://0.0.0.0:4567') turning message relay requesting off
20:48:12 localhost rsyncd[6403]: connect from bb-db-device-01 (10.150.28.18)
20:48:12 localhost rsyncd[6404]: connect from bb-db-device-01 (10.150.28.18)
20:48:12 localhost rsyncd[6310]: sent 63 bytes received 1,074,004,192 bytes total size 1,073,741,824
20:48:12 localhost rsyncd[6404]: rsync to rsync_sst/./performance_schema from bb-db-device-01 (10.150.28.18)
20:48:12 localhost rsyncd[6403]: rsync to rsync_sst/./mysql from bb-db-device-01 (10.150.28.18)
20:48:12 localhost rsyncd[6404]: receiving file list
20:48:12 localhost rsyncd[6403]: receiving file list
20:48:12 localhost rsyncd[6407]: connect from bb-db-device-01 (10.150.28.18)
20:48:12 localhost rsyncd[6404]: sent 1,036 bytes received 100,851 bytes total size 96,357
20:48:12 localhost rsyncd[6403]: sent 1,682 bytes received 1,036,830 bytes total size 1,030,397
20:48:12 localhost rsyncd[6407]: rsync to rsync_sst/ from bb-db-device-01 (10.150.28.18)
20:48:12 localhost rsyncd[6407]: receiving file list
20:48:12 localhost mysqld: 150813 20:48:12 [Note] WSREP: 0.0 (avc2-wus1-db04): State transfer to 1.0 (avc2-wus1-db05) complete.
20:48:12 localhost mysqld: 150813 20:48:12 [Note] WSREP: Member 0.0 (avc2-wus1-db04) synced with group.
20:48:12 localhost rsyncd[6407]: sent 48 bytes received 183 bytes total size 39
20:48:12 localhost mysqld: WSREP_SST: [INFO] Extracting binlog files: (20150813 20:48:12.683)
20:48:12 localhost mysqld: mariadb-bin.000014
20:48:12 localhost mysqld: WSREP_SST: [INFO] Joiner cleanup. (20150813 20:48:12.693)
20:48:13 localhost rsyncd[6293]: sent 0 bytes received 0 bytes total size 0
20:48:13 localhost mysqld: WSREP_SST: [INFO] Joiner cleanup done. (20150813 20:48:13.201)
20:48:13 localhost mysqld: 150813 20:48:13 [Note] WSREP: SST complete, seqno: 2
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Using mutexes to ref count buffer pool pages
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: The InnoDB memory heap is disabled
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Memory barrier is not used
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Compressed tables use zlib 1.2.8
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Using Linux native AIO
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Using CPU crc32 instructions
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Initializing buffer pool, size = 10.0G
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Completed initialization of buffer pool
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Highest supported file format is Barracuda.
20:48:13 localhost mysqld: 150813 20:48:13 [Warning] InnoDB: The log sequence number in the ibdata files is higher than the log sequence number in the ib_logfiles! Are you sure you are using the right ib_logfiles to start up the database. Log sequence number in the ib_logfiles is 1600939, logsequence numbers stamped to ibdata file headers are between 1601220 and 1601220.
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: The log sequence numbers 1601220 and 1601220 in ibdata files do not match the log sequence number 1600939 in the ib_logfiles!
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Database was not shutdown normally!
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Starting crash recovery.
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Reading tablespace information from the .ibd files...
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: Restoring possible half-written data pages
20:48:13 localhost mysqld: 150813 20:48:13 [Note] InnoDB: from the doublewrite buffer...
20:48:16 localhost /etc/init.d/mysql[6553]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
20:48:16 localhost /etc/init.d/mysql[6553]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
20:48:16 localhost /etc/init.d/mysql[6553]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")'
20:48:16 localhost /etc/init.d/mysql[6553]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
20:48:16 localhost /etc/init.d/mysql[6553]:
20:48:27 localhost mysqld: 2015-08-13 20:48:27 7f7ad3fcb7c0 InnoDB: Error: page 5 log sequence number 1601220
20:48:27 localhost mysqld: InnoDB: is in the future! Current system log sequence number 1600939.
20:48:27 localhost mysqld: InnoDB: Your database may be corrupt or you may have copied the InnoDB
20:48:27 localhost mysqld: InnoDB: tablespace but not the InnoDB log files. See
20:48:27 localhost mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
20:48:27 localhost mysqld: InnoDB: for more information.
20:48:28 localhost mysqld: 150813 20:48:28 [Warning] InnoDB: Resizing redo log from 2*32768 to 2*131008 pages, LSN=1600939
20:48:28 localhost mysqld: 150813 20:48:28 [Warning] InnoDB: Starting to delete and rewrite log files.
20:48:29 localhost mysqld: 150813 20:48:29 [Note] InnoDB: Setting log file /mysql_logdir/ib_logfile101 size to 2047 MB
20:48:29 localhost mysqld: 150813 20:48:29 [Note] InnoDB: Setting log file /mysql_logdir/ib_logfile1 size to 2047 MB
20:48:29 localhost mysqld: 150813 20:48:29 [Note] InnoDB: Renaming log file /mysql_logdir/ib_logfile101 to /mysql_logdir/ib_logfile0
20:48:29 localhost mysqld: 150813 20:48:29 [Warning] InnoDB: New log files created, LSN=1601036
20:48:29 localhost mysqld: 150813 20:48:29 [Note] InnoDB: 128 rollback segment(s) are active.
20:48:29 localhost mysqld: 150813 20:48:29 [Note] InnoDB: Waiting for purge to start
20:48:29 localhost mysqld: 150813 20:48:29 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.24-72.2 started; log sequence number 1600939
20:48:29 localhost mysqld: 150813 20:48:29 [Note] Plugin 'FEEDBACK' is disabled.
20:48:29 localhost mysqld: 150813 20:48:29 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
20:48:29 localhost mysqld: 150813 20:48:29 [Note] Starting crash recovery...
20:48:29 localhost mysqld: 150813 20:48:29 [Note] Crash recovery finished.
20:48:29 localhost mysqld: 150813 20:48:29 [Warning] InnoDB: Cannot open table mysql/gtid_slave_pos from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
20:48:29 localhost mysqld: 150813 20:48:29 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1932: Table 'mysql.gtid_slave_pos' doesn't exist in engine
20:48:29 localhost mysqld: 150813 20:48:29 [Note] Server socket created on IP: '0.0.0.0'.
20:48:29 localhost mysqld: 150813 20:48:29 [Warning] 'user' entry 'root@avc2-wus1-db04' ignored in --skip-name-resolve mode.
20:48:29 localhost mysqld: 150813 20:48:29 [Warning] 'proxies_priv' entry '@% root@avc2-wus1-db04' ignored in --skip-name-resolve mode.
20:48:29 localhost mysqld: 150813 20:48:29 [Note] Event Scheduler: Loaded 0 events
20:48:29 localhost mysqld: 150813 20:48:29 [Note] WSREP: Signalling provider to continue.
20:48:29 localhost mysqld: 150813 20:48:29 [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:2
20:48:29 localhost mysqld: 150813 20:48:29 [ERROR] WSREP: Application received wrong state:
20:48:29 localhost mysqld: #011Received: 00000000-0000-0000-0000-000000000000
20:48:29 localhost mysqld: #011Required: 76a27c1d-41f1-11e5-846a-73bafbb95904
20:48:29 localhost mysqld: 150813 20:48:29 [ERROR] WSREP: Application state transfer failed. This is unrecoverable condition, restart required.
20:48:29 localhost mysqld: 150813 20:48:29 [Note] WSREP: Closing send monitor...
20:48:29 localhost mysqld: 150813 20:48:29 [Note] WSREP: Closed send monitor.
20:48:29 localhost mysqld: 150813 20:48:29 [Note] WSREP: gcomm: terminating thread
20:48:29 localhost mysqld: 150813 20:48:29 [Note] WSREP: gcomm: joining thread
20:48:29 localhost mysqld: 150813 20:48:29 [Warning] InnoDB: Cannot open table mysql/gtid_slave_pos from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
20:48:29 localhost mysqld: 150813 20:48:29 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1932: Table 'mysql.gtid_slave_pos' doesn't exist in engine
20:48:29 localhost mysqld: 150813 20:48:29 [Note] WSREP: gcomm: closing backend
20:48:29 localhost mysqld: 150813 20:48:29 [Note] Reading of all Master_info entries succeded
20:48:29 localhost mysqld: 150813 20:48:29 [Note] Added new Master_info '' to hash table
20:48:29 localhost mysqld: 150813 20:48:29 [Note] /usr/sbin/mysqld: ready for connections.
20:48:29 localhost mysqld: Version: '10.0.20-MariaDB-1~trusty-wsrep-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution, wsrep_25.10.r4144
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: view(view_id(NON_PRIM,620d451d,4) memb {
20:48:30 localhost mysqld: #0118e8c1fdb,0
20:48:30 localhost mysqld: } joined {
20:48:30 localhost mysqld: } left {
20:48:30 localhost mysqld: } partitioned {
20:48:30 localhost mysqld: #011620d451d,0
20:48:30 localhost mysqld: })
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: view((empty))
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: gcomm: closed
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: Flow-control interval: [16, 16]
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: Received NON-PRIMARY.
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: Shifting JOINER -> OPEN (TO: 2)
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: Received self-leave message.
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: Flow-control interval: [0, 0]
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: Received SELF-LEAVE. Closing connection.
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 2)
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: RECV thread exiting 0: Success
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: recv_thread() joined.
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: Closing replication queue.
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: Closing slave action queue.
20:48:30 localhost mysqld: 150813 20:48:30 [Note] WSREP: /usr/sbin/mysqld: Terminated.
20:48:30 localhost mysqld_safe: Number of processes running now: 0
20:48:30 localhost mysqld_safe: WSREP: not restarting wsrep node automatically
20:48:30 localhost mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
If you have any ideas I would really appreciate it.
Thanks, Aaron
Lösung
I did manage to get them to join by commenting out these three lines on each server:
innodb_log_group_home_dir = /mysql_logdir
innodb_data_home_dir = /mysql_datadir
wsrep_provider_options = "gcache.size=300M; gcache.page_size=1G"
tried it all again, and all three joined without issue. Then I re-instated those three lines, restarted each node one by one, and I have my cluster..... This is very weird to me, it's like there is something that was timing out in the server, that prevented it from forming the cluster. But once the cluster was up,
Lizenziert unter: CC-BY-SA mit Zuschreibung
Nicht verbunden mit dba.stackexchange
