MySQL Cluster - Management Node Redundancy

Question

I am trying to design a fully redundant MySQL Cluster implementation for a service provider's network. Nodes within this cluster will be geographically separated between towns in the state that the service provider resides. Though I understand that using Geographic Replication for MySQL Cluster requires an asynchronous replication configuration, since the two data centers that the cluster nodes will reside in are connected via a high bandwidth (10Gbps), low latency pipe (~2ms), my goal is to have geographic diversity and synchronous replication across these data centers.

The nodes per data center break down like this:

Data Center A:

• Management Node 1
• SQL(API) Nodes 1 and 2
• Data Node 1 (Node Group 1) and Data Node 2 (Node Group 2)

Data Center B:

• Management Node 2
• SQL Node 3
• Data Node 3 (Node Group 1) and Data Node 4 (Node Group 2)

In this scenario, data node redundancy is strong (I believe), considering that a node from each node group will reside at each data center and a given data node in node group 1 (in either data center A or B) will have half of the data while the data node from the other node group in this same data center will have the other half of the data. In the instance of an isolation event or other type of failure that breaks apart the data centers, each location will have at least a full copy of the data.

My concern now becomes having a management node located in each data center. I am not sure if redundancy for the management nodes is best implemented this way or not. From what I understand about MySQL Cluster, if an isolation event occurs at one of the data centers (let's say data center A) and the connection between these two locations is lost, data nodes in data center B would start looking at management node 2 in this data center as the arbitrator, instead of the management node 1 in data center A, which would have originally been the arbitrator.

In my mind, this situation creates a split brain operation, where each data center is looking at its own arbitrator and, as a result, you still have all data nodes turned up in the cluster, yet only half of the storage locations are being written to. You break your synchronous environment.

Should I instead have the redundant management node also located in data center A on a separate physical resource, or does this change nothing about the logic of the cluster? Should I go ahead and instead create an asynchronous replication environment for the database and have each data center house its own cluster instead of breaking a cluster out between two geographic locations?

Warning: This is my first database project ever, so I am quite new to this stuff. I may be completely butchering this design implementation.

Answer 1

If you have split exactly in half brain situation MySQL Cluster resolves it designating one node as master. If you look into ndb_mgm output below you will notice asterisk at the end of node 10 line. It marks master.

It means that if your cluster will be split in two equal parts the half which contains master will continue running. Nodes from the other half will be shut down.

ndb_mgm> show
Connected to Management Server at: 10.1.1.10:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=10   @10.1.1.20  (mysql-5.6.24 ndb-7.3.9, Nodegroup: 0, *)
id=11   @10.1.1.21  (mysql-5.6.24 ndb-7.3.9, Nodegroup: 0)