Multi-Master Multi-Region MySQL Cluster

Question

I am hosting an application (a game) in multiple regions around the globe. Performance is critical to this application since it's a game, and therefore I need to host a database instance in each of these locations that will serve the local application instances.
Due to the nature of my application, I am only able to use MySQL.

My main requirement is that each of these instances will be writable. I understand that in most cases this is a big problem in MySQL clusters when it comes to two servers modifying the same record.
However, in my application this is actually impossible; a player can only be connected to a single region at any given moment, which means only that region's database instance will read or write that player's data. Consistency is also not critical here and Eventual Consistency is okay for my needs. I just need to make sure it gets there eventually.

I haven't worked with replication before and therefore I need some guidance. What would be the right way to implement this, and how would I go about deploying such a cluster?

Answer 1

You probably want a cluster such as MySQL InnoDB Cluster (from Oracle) or MariaDB Galera Cluster or Percona XtraDB Cluster in each region for high availability. You can write to any of the nodes in the cluster and the transaction will be automatically replicated to the other nodes. Note that both Percona XtraDB Cluster and MariaDB Galera Cluster are based on the Galera cluster technology. Galera also provides libraries for regular MySQL.

For an automatic failover solution, you may want to use one of these solutions:

1. A library in your application that detects whether a node is down, and if so, attempts to use another node in the cluster. This can be e.g. MariaDB's JDBC driver - MariaDB connector/J which I believe also works with Percona and MySQL. You need to configure the connector to use a cluster.
2. A database proxy for each cluster/region, such as ProxySQL (GPL) or MaxScale (BSL, but older versions automatically become GPL - currently the newest GPL version is 2.1). Database proxies are database-aware: If there's a problem with any of the nodes, it will mark that node as unavailable until the problem has been resolved.

And then you probably want to have master-master asynchronous replication between those clusters. Master-master asynchronous replication simply means that you set up asynchronous master-slave replication from node A to node B and at the same time also asynchronous master-slave replication from node B to node A.

Note that asynchronous master-master replication is notoriously problematic if there is a risk that the two nodes will write to the same rows. Many tutorials on the topic also fail to mention how to avoid many collision problems by setting different values for auto_increment_increment and auto_increment_offset for each node.

Also note that geo-distributed nodes within a cluster is probably not a good idea due to latency and unreliable networks, although Galera doesn't seem to be concerned about this issue in their documentation.

Personally, I wouldn't want to use MySQL 8.0 because of inherent stability issues with the new development and release model: Major new features are now introduced in patch releases (8.0.x). Everyone obviously wants to use the latest patch release because it contains bug & security fixes of previous versions, but since the patch releases also contain major new features, there is a serious risk that new bugs will also be introduced. I'm also not sure what free options you have for non-intrusive backup with MySQL 8.0. (There are paid options, of course, if you go with a MySQL Enterprise license.) You may be able to use Percona Xtrabackup.

It's also possible there is a cloud DBaaS solution for this use-case such as AWS RDS or MariaDB Platform Managed Database Service. I'm unsure how this compares in terms of cost and features, but it might be considerably easier to configure and manage than what I have outlined above.

Answer 2

"Eventual consistency" is the keystone of MySQL's "NDB Cluster". Each node can be "immediately" written to.

NDB is distinct from Galera/InnoDB/Percona clusters.