Redis-Sentinel stuck with slaveof on each node

StackOverflow https://stackoverflow.com/questions/18855841

Domanda

i'm using redis-sentinel from the 2.8 RC5 of redis to monitor and fail-over my redis nodes. The setup is as follows:

Sentinel nodes

  • Server A
  • Server B
  • Server C

Redis nodes

  • Server A (Master)
  • Server B (Slave of Server A -- using slaveof in the redis config)

On each sentinel-node i'm using the same config:

sentinel monitor serverA serverA.mydomain.tld 6379 2
sentinel auth-pass serverA "MYAUTHPASS"
sentinel down-after-milliseconds serverA 10000
sentinel failover-timeout serverA 20000
sentinel can-failover serverA yes
sentinel parallel-syncs serverA 1

sentinel monitor serverB serverB.mydomain.tld 6379 4
sentinel auth-pass serverB "MYAUTHPASS"
sentinel down-after-milliseconds serverB 10000
sentinel failover-timeout serverB 20000
sentinel can-failover serverB yes
sentinel parallel-syncs serverB 5

I'm doing this test run:

  1. Start both redis-nodes
  2. Start each sentinel node (all three)
  3. Stop Server A (Master)
  4. Server B becomes the master
  5. Start Server A (Master) again
  6. Stop Server B (Slave)
  7. Start Server B (Slave) again

Now Server A is the slave of Server B and Server A is the slave of Server B.

Is that a bug or is that a mis-configuration?

These are the sentinel logs after Step 3 (Stop the Master on Server A)

[19569] 17 Sep 18:33:28.873 # +sdown master serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:28.873 # +sdown master serverB serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:29.073 # +odown master serverA serverA.mydomain.tld 6379 #quorum 3/2
[19569] 17 Sep 18:33:29.073 # +failover-triggered master serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:29.073 # +failover-state-wait-start master serverA serverA.mydomain.tld 6379 #starting in 6543 milliseconds
[19569] 17 Sep 18:33:35.700 # +failover-state-select-slave master serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:35.800 # +selected-slave slave serverB.mydomain.tld:6379 serverB.mydomain.tld 6379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:35.800 * +failover-state-send-slaveof-noone slave serverB.mydomain.tld:6379 serverB.mydomain.tld 6379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:35.900 * +failover-state-wait-promotion slave serverB.mydomain.tld:6379 serverB.mydomain.tld 6379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:36.204 # +promoted-slave slave serverB.mydomain.tld:6379 serverB.mydomain.tld 6379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:36.204 # +failover-state-reconf-slaves master serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:36.302 # +failover-end master serverA serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:36.302 # +switch-master serverA serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:37.196 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:38.107 # +failover-detected master serverB serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:38.206 # +failover-end master serverB serverA.mydomain.tld 6379
[19569] 17 Sep 18:33:38.206 # +switch-master serverB serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:41.322 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:41.322 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:42.105 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:46.329 # +sdown slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:33:48.233 # +sdown slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverB serverB.mydomain.tld 6379

Overview of CONFIG GET slaveof

  • Server A: not online
  • Server B: ""

These are the logs after starting Server A (Master) again (Step 5):

[19569] 17 Sep 18:53:08.305 * +demote-old-slave slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:53:08.306 * +demote-old-slave slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 18:53:08.506 # -sdown slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:53:08.506 # -sdown slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 18:53:18.319 * +slave slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:53:18.319 * +slave slave serverA.mydomain.tld:6379 serverA.mydomain.tld 6379 @ serverB serverB.mydomain.tld 6379

Overview of CONFIG GET slaveof

  • Server A: "serverB.mydomain.tld"
  • Server B: ""

These are the logs after stopping Server B (slave):

[19569] 17 Sep 18:58:38.375 # +sdown master serverB serverB.mydomain.tld 6379
[19569] 17 Sep 18:58:38.675 # +sdown master serverA serverB.mydomain.tld 6379
[19569] 17 Sep 18:58:38.876 # +odown master serverA serverB.mydomain.tld 6379 #quorum 3/2

Overview of CONFIG GET slaveof

  • Server A: "serverB.mydomain.tld" (seems to be wrong at this time!)
  • Server B: not online

These are the logs after starting Server B (slave) again:

[19569] 17 Sep 19:00:59.892 * +reboot master serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:00:59.892 # +redirect-to-master serverB serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:00:59.892 * +reboot master serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:00:59.892 # +redirect-to-master serverA serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:00.012 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:00.012 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:05.008 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:05.008 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:09.907 # +redirect-to-master serverA serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:09.907 # +redirect-to-master serverB serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:10.029 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:10.029 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:10.046 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:10.046 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:19.925 # +redirect-to-master serverA serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:19.925 # +redirect-to-master serverB serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:20.096 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:20.096 * +sentinel sentinel serverC.mydomain.tld:263
[19569] 17 Sep 19:01:20.096 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:20.096 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:20.143 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:20.143 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:29.943 # +redirect-to-master serverA serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:29.943 # +redirect-to-master serverB serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:30.037 - Reading from client: Connection reset by peer
[19569] 17 Sep 19:01:30.140 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:30.140 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:30.156 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:30.156 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:39.962 # +redirect-to-master serverA serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:39.962 # +redirect-to-master serverB serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:40.169 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:40.169 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:40.191 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:40.191 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:49.980 # +redirect-to-master serverA serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:49.980 # +redirect-to-master serverB serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:50.180 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:50.180 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:50.256 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:50.256 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:01:59.999 # +redirect-to-master serverA serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:01:59.999 # +redirect-to-master serverB serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:02:00.193 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:02:00.193 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:02:00.313 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:02:00.313 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:02:10.013 # +redirect-to-master serverA serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:02:10.013 # +redirect-to-master serverB serverA.mydomain.tld 6379 serverB.mydomain.tld 6379
[19569] 17 Sep 19:02:10.209 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:02:10.209 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379
[19569] 17 Sep 19:02:10.450 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverB.mydomain.tld 6379
[19569] 17 Sep 19:02:10.450 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverB.mydomain.tld 6379

Overview of CONFIG GET slaveof

  • Server A: "serverB.mydomain.tld" (now this is wrong!)
  • Server B: "serverA.mydomain.tld" (now this is wrong!)

Additionaly, this section repeats every 5 seconds (endlessly):

[19569] 17 Sep 19:05:40.351 # +redirect-to-master serverA serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:05:40.351 # +redirect-to-master serverB serverB.mydomain.tld 6379 serverA.mydomain.tld 6379
[19569] 17 Sep 19:05:40.723 - Accepted serverB.mydomain.tld:38423
[19569] 17 Sep 19:05:40.724 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:05:40.724 - Accepted serverB.mydomain.tld:38424
[19569] 17 Sep 19:05:40.724 * +sentinel sentinel serverB.mydomain.tld:26379 serverB.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
[19569] 17 Sep 19:05:41.514 - Client closed connection
[19569] 17 Sep 19:05:41.515 - Client closed connection
[19569] 17 Sep 19:05:42.112 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverA serverA.mydomain.tld 6379
[19569] 17 Sep 19:05:42.112 * +sentinel sentinel serverC.mydomain.tld:26379 serverC.mydomain.tld 26379 @ serverB serverA.mydomain.tld 6379
È stato utile?

Soluzione

You have to specify only the master server in your sentinel configuration (only masters are specified there) as the slave(s) should be automatically discovered by sentinel itself.

It is stucking as both here are considered as masters in your sentinel config.

I would recommend reading the redis-sentinel documentation to get more info and how it is working

Altri suggerimenti

I would modify your test procedure as follows (the bold lines are added):

  1. Start both redis-nodes
  2. start each sentinel node (all three)
  3. Stop Server A (Master)
  4. Server B becomes the master
  5. Start Server A (Master) again
  6. on A, run slaveof(B)
  7. wait for sentinel cluster to detect A
  8. Stop Server B (Slave)
  9. Start Server B (Slave) again
  10. on B, run slaveof(A)

If you're specifically trying to test a degenerate case, definitely ignore my advice. But if you're trying to simulate a real recovery scenario, the boldface lines should tell the sentinel cluster what's up.

Autorizzato sotto: CC-BY-SA insieme a attribuzione
Non affiliato a StackOverflow
scroll top