MySQL Cluster ERROR 1296 (HY000): Got error 157 'Unknown error code' from NDBCLUSTER

Question

Today my datacenter had a breaker fail which resulted in my servers losing power. I'm running a 4 node MySQL cluster. I restarted the cluster, first the management nodes, then the data nodes, then after the data nodes were running I started the SQL nodes. I then checked the cluster with ndb_mgm -e SHOW. Everything seemed fine until I tried running a query. I got this error,

ERROR 1296 (HY000): Got error 157 'Unknown error code' from NDBCLUSTER

I check the MySQL logs and could not find any errors. I then tried a full shutdown and restart of the MySQL cluster and checked config between the shutdown and start. Everything seemed to check out. I then ran a query on another database using the NDBCLUSTER engine. The query was successful. I've tried searching google but no one seems to have any answers that help. I've checked the config, I've made sure ndbd is running on the data nodes, ect. The other databases seem to be working fine except this one. I have a backup of the database but I would much preferably recover the database if possible.

If anyone has any suggestion or ideas, it would be greatly appreciated.

Thanks in advance.

Answer 1

Error 157 is actually 'could not connect to storage engine' and the fact that MySQL fails to report that error correctly is a bug: http://bugs.mysql.com/bug.php?id=44817

The case described in that bug mentions that you get the error when you try to query a table in NDB when the cluster is still down.

So I'm just guessing, but I would conclude that your cluster is not started. Either you missed starting one of the nodes, or else something went wrong starting one of the nodes.

Answer 2

Check the mysql server is really connected to the NDB storage. Do from a mysql server that should be connected to NDB

SHOW GLOBAL STATUS LIKE 'Ndb_cluster_node_id';

Is the answer > 0?

SHOW GLOBAL STATUS LIKE 'Ndb_number_of_data_nodes';

Is the answer > 0 ?

If not, then the mysql server is not connected and then i would recommend you check your firewall and /etc/hosts table and make sure you dont have a line like:

127.0.0.1 localhost ..

Best regards

Johan