MariaDB 10.1.16 on Ubuntu fails to start: WSREP Failed to recover position

Question

I'm using 10.1.16-MariaDB-1~xenial from the official MariaDB apt repository for 10.1 [stable], via the University of Texas mirror.

I had a perfectly functioning MariaDB Galera cluster setup on 3 Ubuntu 16.04 servers.

Then I upgraded them. Now I have nothing.

The upgrade to 10.1.16 failed, and quickly brought down the whole cluster. I don't have the output, but dpkg failed on setting up mariadb-server and mariadb-server-10.1.

I have backups, so I purged all traces of MariaDB/MySQL/Galera from my servers (including removing /var/lib/mysql/, /etc/mysql/, and /var/log/mysql/) and started over. However, now, with a clean install on each server, none of the standard system startup scripts work. I suspect this is why the upgrade process through apt failed, too.

I've tried each of the following on my first node:

galera_new_cluster
service mysql bootstrap
service mysql bootstrap --wsrep-new-cluster
service mysql bootstrap --wsrep-cluster-address="gcomm://"
service mysql start
service mysql start --wsrep-new-cluster
service mysql start --wsrep-cluster-address="gcomm://"
systemctl start mariadb
systemctl start mariadb --wsrep-new-cluster
systemctl start mariadb --wsrep-cluster-address="gcomm://"

Every single one gives me the same output:

Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.

systemctl status mariadb.service:

● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: failed (Result: exit-code) since Fri 2016-07-22 13:29:45 CDT; 42s ago
  Process: 10799 ExecStartPre=/bin/sh -c VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] &&   systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=1/FAILURE)
  Process: 10794 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 Main PID: 16865 (code=exited, status=0/SUCCESS)

Jul 22 13:29:41 sql2 systemd[1]: Starting MariaDB database server...
Jul 22 13:29:45 sql2 mysqld[10799]: WSREP: Failed to recover position: '2016-07-22 13:29:41 140110745778432 [Note] /usr/sbin/mysqld (mysqld 10.1.16-MariaDB-1~xenial) starting as process 11080 ...'
Jul 22 13:29:45 sql2 systemd[1]: mariadb.service: Control process exited, code=exited status=1
Jul 22 13:29:45 sql2 systemd[1]: Failed to start MariaDB database server.
Jul 22 13:29:45 sql2 systemd[1]: mariadb.service: Unit entered failed state.
Jul 22 13:29:45 sql2 systemd[1]: mariadb.service: Failed with result 'exit-code'.

The only way I can start my servers now is by manually executing:

sudo -u mysql mysqld --wsrep-cluster-address='gcomm://'

On the first node, and then:

sudo -u mysql mysqld --wsrep-cluster-address='gcomm://ip1,ip2,ip3'

On the other two nodes. That works, and I have a working cluster again. But now, systemd/systemctl have no idea the service is running. It seems like the systemd startup scripts can't use the wsrep-cluster-address setting in my configuration files at all. Specifying it to service or systemctl command line does not work either.

How am I supposed to start mariadb?

Answer 1

There was a bug in galera_recovery.sh script. https://jira.mariadb.org/browse/MDEV-10396

Answer 2

I've been able to temporarily alleviate my problems by downgrading to 10.1.14:

wget https://downloads.mariadb.com/files/MariaDB/mariadb-10.1.14/repo/ubuntu/pool/main/m/mariadb-10.1/mariadb-server-10.1_10.1.14+maria-1~xenial_amd64.deb
wget https://downloads.mariadb.com/files/MariaDB/mariadb-10.1.14/repo/ubuntu/pool/main/m/mariadb-10.1/libmariadbclient18_10.1.14+maria-1~xenial_amd64.deb
wget https://downloads.mariadb.com/files/MariaDB/mariadb-10.1.14/repo/ubuntu/pool/main/m/mariadb-10.1/libmysqlclient18_10.1.14+maria-1~xenial_amd64.deb
wget https://downloads.mariadb.com/files/MariaDB/mariadb-10.1.14/repo/ubuntu/pool/main/m/mariadb-10.1/mariadb-client-10.1_10.1.14+maria-1~xenial_amd64.deb
wget https://downloads.mariadb.com/files/MariaDB/mariadb-10.1.14/repo/ubuntu/pool/main/m/mariadb-10.1/mariadb-client-core-10.1_10.1.14+maria-1~xenial_amd64.deb
wget https://downloads.mariadb.com/files/MariaDB/mariadb-10.1.14/repo/ubuntu/pool/main/m/mariadb-10.1/mariadb-common_10.1.14+maria-1~xenial_all.deb
wget https://downloads.mariadb.com/files/MariaDB/mariadb-10.1.14/repo/ubuntu/pool/main/m/mariadb-10.1/mariadb-server-core-10.1_10.1.14+maria-1~xenial_amd64.deb
wget https://downloads.mariadb.com/files/MariaDB/mariadb-10.1.14/repo/ubuntu/pool/main/m/mariadb-10.1/mysql-common_10.1.14+maria-1~xenial_all.deb
wget https://downloads.mariadb.com/files/MariaDB/mariadb-10.1.14/repo/ubuntu/pool/main/g/galera-3/galera-3_25.3.15-xenial_amd64.deb
apt purge mariadb-server
apt autoremove --purge
apt clean
rm -rf /etc/mysql/
rm -rf /var/lib/mysql/
rm -rf /var/log/mysql/
rm -f /var/log/mysql.*
rm -f /etc/systemd/system/mysql.service
rm -f /etc/systemd/system/mysqld.service
rm -f /etc/rc0.d/K03mysql
rm -f /etc/rc1.d/K03mysql
rm -f /etc/rc2.d/S03mysql
rm -f /etc/rc3.d/S03mysql
rm -f /etc/rc4.d/S03mysql
rm -f /etc/rc5.d/S03mysql
rm -f /etc/rc6.d/K03mysql
rm -f /var/lib/systemd/deb-systemd-helper-enabled/mysql.service
rm -f /var/lib/systemd/deb-systemd-helper-enabled/mysqld.service
rm -f /etc/apparmor.d/abstractions/mysql
rm -f /etc/apparmor.d/cache/usr.sbin.mysqld
rm -rf /etc/systemd/system/mariadb.service.d
rm -f /etc/systemd/system/multi-user.target.wants/mariadb.service
rm -f /var/lib/systemd/deb-systemd-helper-enabled/mariadb.service.dsh-also
rm -f /var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants/mariadb.service
rm -f /var/lib/systemd/deb-systemd-helper-enabled/mysql.service
rm -f /var/lib/systemd/deb-systemd-helper-enabled/mysqld.service
rm -rf /var/lib/apt/lists/*
apt update
apt install iproute libaio1 libcgi-fast-perl libcgi-pm-perl libdbd-mysql-perl libdbi-perl libencode-locale-perl libfcgi-perl libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl libhttp-date-perl libhttp-message-perl libio-html-perl libjemalloc1 liblwp-mediatypes-perl libtimedate-perl liburi-perl socat
dpkg -i mysql-common_10.1.14+maria-1~xenial_all.deb 
dpkg -i mariadb-common_10.1.14+maria-1~xenial_all.deb 
dpkg -i libmariadbclient18_10.1.14+maria-1~xenial_amd64.deb libmysqlclient18_10.1.14+maria-1~xenial_amd64.deb mariadb-client-10.1_10.1.14+maria-1~xenial_amd64.deb mariadb-client-core-10.1_10.1.14+maria-1~xenial_amd64.deb mariadb-server-10.1_10.1.14+maria-1~xenial_amd64.deb mariadb-server-core-10.1_10.1.14+maria-1~xenial_amd64.deb galera-3_25.3.15-xenial_amd64.deb 

Now, I can start my first node with galera_new_cluster and all other nodes with service mysql start.

There must be a bug with the systemd/systemctl scripts in 10.1.16.