i'm using redis-sentinel
from the 2.8 RC5 of redis to monitor and fail-over my redis nodes. The setup is as follows:
Sentinel nodes
- Server A
- Server B
- Server C
Redis nodes
- Server A (Master)
- Server B (Slave of Server A -- using
slaveof
in the redis config)
On each sentinel-node i'm using the same config:
sentinel monitor serverA serverA.mydomain.tld 6379 2
sentinel auth-pass serverA "MYAUTHPASS"
sentinel down-after-milliseconds serverA 10000
sentinel failover-timeout serverA 20000
sentinel can-failover serverA yes
sentinel parallel-syncs serverA 1
sentinel monitor serverB serverB.mydomain.tld 6379 4
sentinel auth-pass serverB "MYAUTHPASS"
sentinel down-after-milliseconds serverB 10000
sentinel failover-timeout serverB 20000
sentinel can-failover serverB yes
sentinel parallel-syncs serverB 5
I'm doing this test run:
- Start both redis-nodes
- Start each sentinel node (all three)
- Stop Server A (Master)
- Server B becomes the master
- Start Server A (Master) again
- Stop Server B (Slave)
- Start Server B (Slave) again
Now Server A is the slave of Server B and Server A is the slave of Server B.
Is that a bug or is that a mis-configuration?
These are the sentinel logs after Step 3 (Stop the Master on Server A)
[19569] 17 Sep 18:33:28.873 # +sdown master serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:28.873 # +sdown master serverB serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:29.073 # +odown master serverA serverA.mydomain.tld 6379 #quorum 3/2
[19569] 17 Sep 18:33:29.073 # +failover-triggered master serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:29.073 # +failover-state-wait-start master serverA serverA.mydomain.tld 6379 #starting in 6543 milliseconds
[19569] 17 Sep 18:33:35.700 # +failover-state-select-slave master serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:35.800 # +selected-slave slave serverB.mydomain.tld:6379 serverB.mydomain.tld 6379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:35.800 * +failover-state-send-slaveof-noone slave serverB.mydomain.tld:6379 serverB.mydomain.tld 6379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:35.900 * +failover-state-wait-promotion slave serverB.mydomain.tld:6379 serverB.mydomain.tld 6379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:36.204 # +promoted-slave slave serverB.mydomain.tld:6379 serverB.mydomain.tld 6379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:36.204 # +failover-state-reconf-slaves master serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:36.302 # +failover-end master serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:36.302 # +switch-master serverA serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:37.196 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:38.107 # +failover-detected master serverB serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:38.206 # +failover-end master serverB serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:38.206 # +switch-master serverB serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:41.322 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:41.322 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:42.105 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:46.329 # +sdown slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:48.233 # +sdown slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverB serverB.mydomain.tld 6379
Overview of CONFIG GET slaveof
- Server A: not online
- Server B:
""
These are the logs after starting Server A (Master) again (Step 5):
[19569] 17 Sep 18:53:08.305 * +demote-old-slave slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:53:08.306 * +demote-old-slave slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 18:53:08.506 # -sdown slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:53:08.506 # -sdown slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 18:53:18.319 * +slave slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:53:18.319 * +slave slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverB serverB.mydomain.tld 6379
Overview of CONFIG GET slaveof
- Server A:
"serverB.mydomain.tld"
- Server B:
""
These are the logs after stopping Server B (slave):
[19569] 17 Sep 18:58:38.375 # +sdown master serverB serverB.mydomain.tld 6379
[19569] 17 Sep 18:58:38.675 # +sdown master serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:58:38.876 # +odown master serverA serverB.mydomain.tld 6379 #quorum 3/2
Overview of CONFIG GET slaveof
- Server A:
"serverB.mydomain.tld"
(seems to be wrong at this time!)
- Server B: not online
These are the logs after starting Server B (slave) again:
[19569] 17 Sep 19:00:59.892 * +reboot master serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:00:59.892 # +redirect-to-master serverB serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:00:59.892 * +reboot master serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:00:59.892 # +redirect-to-master serverA serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:00.012 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:00.012 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:05.008 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:05.008 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:09.907 # +redirect-to-master serverA serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:09.907 # +redirect-to-master serverB serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:10.029 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:10.029 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:10.046 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:10.046 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:19.925 # +redirect-to-master serverA serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:19.925 # +redirect-to-master serverB serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:20.096 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:20.096 * +sentinel sentinel serverC.mydomain.tld:263
[19569] 17 Sep 19:01:20.096 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:20.096 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:20.143 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:20.143 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:29.943 # +redirect-to-master serverA serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:29.943 # +redirect-to-master serverB serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:30.037 - Reading from client: Connection reset by peer
[19569] 17 Sep 19:01:30.140 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:30.140 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:30.156 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:30.156 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:39.962 # +redirect-to-master serverA serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:39.962 # +redirect-to-master serverB serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:40.169 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:40.169 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:40.191 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:40.191 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:49.980 # +redirect-to-master serverA serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:49.980 # +redirect-to-master serverB serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:50.180 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:50.180 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:50.256 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:50.256 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:59.999 # +redirect-to-master serverA serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:59.999 # +redirect-to-master serverB serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:02:00.193 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:02:00.193 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:02:00.313 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:02:00.313 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:02:10.013 # +redirect-to-master serverA serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:02:10.013 # +redirect-to-master serverB serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:02:10.209 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:02:10.209 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:02:10.450 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:02:10.450 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
Overview of CONFIG GET slaveof
- Server A:
"serverB.mydomain.tld"
(now this is wrong!)
- Server B:
"serverA.mydomain.tld"
(now this is wrong!)
Additionaly, this section repeats every 5 seconds (endlessly):
[19569] 17 Sep 19:05:40.351 # +redirect-to-master serverA serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:05:40.351 # +redirect-to-master serverB serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:05:40.723 - Accepted serverB.mydomain.tld:38423
[19569] 17 Sep 19:05:40.724 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:05:40.724 - Accepted serverB.mydomain.tld:38424
[19569] 17 Sep 19:05:40.724 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:05:41.514 - Client closed connection
[19569] 17 Sep 19:05:41.515 - Client closed connection
[19569] 17 Sep 19:05:42.112 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:05:42.112 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379