Unable to demote DB with repmgr

Question

I am using repmgr and had successfully failed over to a standby node. However when I attempeted to recover the failed master node, I now find that I have two master nodes.

I have tried to re register the standby node as standby. But I find that when repmgr checks the mode of the node, It is finding that it is in recovery mode and does not allow it.

repmgr -f /var/lib/pgsql/repmgr/repmgr.conf cluster show
=> Role      | Connection String 
* master  | host=<IP Address> user=repmgr dbname=my_db
* master  | host=<IP Address> user=repmgr dbname=my_db

Node 1

repmgr -f /var/lib/pgsql/repmgr/repmgr.conf --verbose master register --force
=> Opening configuration file: /var/lib/pgsql/repmgr/repmgr.conf
repmgr connecting to master database
repmgr connected to master, checking its state
finding node list for cluster 'my_cluster'
checking role of cluster node 'host=<IP Address> user=repmgr dbname=my_db'
There is a master already in cluster my_cluster

Node 2
============
repmgr -f /var/lib/pgsql/repmgr/repmgr.conf standby clone -h <master Node IP> --force
=> repmgr: directory "." exists but is not empty

Any advice would be appreciated.

Answer 1

Trick is to switch off the postgres daemon on the server you want to demote. You don't need to switch it back on again until you have completed the clone, by that time cluster show will show that the old master has been successfully demoted.

Depending on your connection pooling logic, you probably want the old master switched off just in-case things get written to it and then lost during the clone anyway.