DEV Community

Ježek
Ježek

Posted on • Edited on

Set up a Redis Sentinel

#redis

Redis® Sentinel provides high availability for Redis. Redis Sentinel also provides other collateral tasks such as monitoring, notifications and acts as a configuration provider for clients.

This article shows how to set up a high-availability architecture for Redis using Sentinel in Debian GNU/Linux:

Redis Sentinel

Prerequisites

Debian preinstalled servers, e.g.:

Host name IP Role
redis-1 10.0.0.21 Master node
redis-2 10.0.0.22 Slave node
redis-3 10.0.0.23 Slave node
sentinel-1 10.0.0.24 Sentinel node
sentinel-2 10.0.0.25 Sentinel node
sentinel-3 10.0.0.26 Sentinel node

Setup Redis replication

Install Redis

  1. Connect to hosts with roles: Master node, Slave node

  2. Run the following commands:

    $ sudo apt-get update
$ sudo apt-get install -y redis-server

  3. Activate the redis-server service to start when Redis server boots:

    $ sudo systemctl enable redis-server

  4. Check redis-server status:

    $ systemctl status redis-server
● redis-server.service - Advanced key-value store
 Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor >
 Active: active (running) since Mon 2024-09-16 21:06:13; 19min ago
   Docs: http://redis.io/documentation,
         man:redis-server(1)
Main PID: 3057 (redis-server)
 Status: "Ready to accept connections"
  Tasks: 5 (limit: 2307)
 Memory: 9.2M
    CPU: 2.146s
 CGroup: /system.slice/redis-server.service
         └─3057 /usr/bin/redis-server 127.0.0.1:6379

    Service runs and ready to accept connections.

Set up the Master node

  1. Open the configuration file at /etc/redis/redis.conf

  2. Set following mandatory parameters:

    protected-mode yes
bind <masterip: 10.0.0.21> 127.0.0.1
requirepass <master-password>
masterauth <master-password>

  3. Restart the redis-server service:

    $ sudo systemctl restart redis-server

  4. Make sure that Redis accepts connections with authentication:

    $ redis-cli --askpass info

Set up the Slave nodes

  1. Open the configuration file at /etc/redis/redis.conf

  2. Set following mandatory parameters:

    bind <slaveip: 10.0.0.21/22> 127.0.0.1
requirepass <master-password>
masterauth <master-password>
replicaof <masterip: 10.0.0.21> 6379

  3. Restart the redis-server service:

    $ sudo systemctl restart redis-server

  4. Make sure that Redis accepts connections with authentication:

    $ redis-cli --askpass info

Check Redis replication setup

  1. Check the redis-server log on Master node:

    $ tail /var/log/redis/redis-server.log -n 20
3498:M 16 Sep 2024 22:25:53.086 * RDB memory usage when created 0.77 Mb
3498:M 16 Sep 2024 22:25:53.086 * DB loaded from disk: 0.000 seconds
3498:M 16 Sep 2024 22:25:53.086 * Ready to accept connections
3498:M 16 Sep 2024 23:29:52.119 * Replica 10.0.0.22:6379 asks for synchronization
3498:M 16 Sep 2024 23:29:52.119 * Full resync requested by replica 10.0.0.22:6379
3498:M 16 Sep 2024 23:29:52.119 * Replication backlog created, my new replication IDs are '368571d1e52c29535615e0043a3a6d6ddc1db70b' and '0000000000000000000000000000000000000000'
3498:M 16 Sep 2024 23:29:52.119 * Starting BGSAVE for SYNC with target: disk
3498:M 16 Sep 2024 23:29:52.119 * Background saving started by pid 3684
3684:C 16 Sep 2024 23:29:52.132 * DB saved on disk
3684:C 16 Sep 2024 23:29:52.133 * RDB: 0 MB of memory used by copy-on-write
3498:M 16 Sep 2024 23:29:52.185 * Background saving terminated with success
3498:M 16 Sep 2024 23:29:52.185 * Synchronization with replica 10.0.0.22:6379 succeeded
3498:M 16 Sep 2024 23:30:09.952 * Replica 10.0.0.23:6379 asks for synchronization
3498:M 16 Sep 2024 23:30:09.952 * Full resync requested by replica 10.0.0.23:6379
3498:M 16 Sep 2024 23:30:09.952 * Starting BGSAVE for SYNC with target: disk
3498:M 16 Sep 2024 23:30:09.952 * Background saving started by pid 3688
3688:C 16 Sep 2024 23:30:09.965 * DB saved on disk
3688:C 16 Sep 2024 23:30:09.965 * RDB: 0 MB of memory used by copy-on-write
3498:M 16 Sep 2024 23:30:09.985 * Background saving terminated with success
3498:M 16 Sep 2024 23:30:09.985 * Synchronization with replica 10.0.0.23:6379 succeeded

  2. Check replication status on Master node:

    $ redis-cli --askpass info replication
Please input password: ****************
# Replication
role:master
connected_slaves:2
slave0:ip=10.0.0.22,port=6379,state=online,offset=1652,lag=1
slave1:ip=10.0.0.23,port=6379,state=online,offset=1652,lag=1
master_replid:368571d1e52c29535615e0043a3a6d6ddc1db70b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1652
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:1652

  3. Run the following command on Master node:

    $ $ redis-cli --askpass set abc 123
Please input password: ****************
OK

  4. Run the following command on Slave nodes:

    $ redis-cli --askpass get abc
Please input password: ****************
"123"

Congrats! Master-to-Slave data replication succeed.

Setup Redis Sentinel

Install Redis Sentinel

  1. Connect to hosts with the role Sentinel

  2. Run the following commands:

    $ sudo apt-get update
$ sudo apt-get install -y redis-sentinel

  3. Activate the redis-sentinel service to start when Sentinel server boots:

    $ sudo systemctl enable redis-sentinel

  4. Check redis-sentinel status:

    $ systemctl status redis-sentinel
● redis-sentinel.service - Advanced key-value store
 Loaded: loaded (/lib/systemd/system/redis-sentinel.service; enabled; vendo>
 Active: active (running) since Tue 2024-09-17 00:56:22 MSK; 1min 29s ago
   Docs: http://redis.io/documentation,
         man:redis-sentinel(1)
Main PID: 3259 (redis-sentinel)
 Status: "Ready to accept connections"
  Tasks: 5 (limit: 2307)
 Memory: 7.2M
    CPU: 190ms
 CGroup: /system.slice/redis-sentinel.service
         └─3259 /usr/bin/redis-sentinel 127.0.0.1:26379 [sentinel]

Set up Sentinel nodes

  1. Open the configuration file at /etc/redis/sentinel.conf

  2. Set following mandatory parameters:

    protected-mode yes
bind <sentinelip: 10.0.0.24/25/26> 127.0.0.1
sentinel monitor mymaster <masterip: 10.0.0.21> 6379 2
sentinel down-after-milliseconds mymaster 30000
sentinel failover-timeout mymaster 180000
sentinel parallel-syncs mymaster 1
sentinel auth-pass mymaster <master-password>
requirepass <sentinel-password>

  3. Restart the redis-sentinel service:

    $ sudo systemctl restart redis-sentinel

  4. Make sure that Redis Sentinel accepts connections with authentication:

    $ redis-cli -p 26379 --askpass info sentinel
Please input password: ****************
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.0.0.21:6379,slaves=2,sentinels=3

Testing the failover

This section shows how to test the failover of the high availability Redis using Sentinel.

Master node failure

  1. Run the following command on Master node:

    $ redis-cli --askpass client pause 40000
Please input password: ****************
OK

    Note: client pause 40000 due to sentinel down-after-milliseconds mymaster 30000

  2. Run the following command on one of the Sentinel nodes:

    $ tail -10 /var/log/redis/redis-sentinel.log
6175:X 18 Sep 2024 21:39:49.309 # +sdown master mymaster 10.0.0.21 6379
6175:X 18 Sep 2024 21:39:49.555 # +new-epoch 1
6175:X 18 Sep 2024 21:39:49.558 # +vote-for-leader 77fc4fbc9c6c7d063d62e38f528910ab67e8d907 1
6175:X 18 Sep 2024 21:39:50.450 # +odown master mymaster 10.0.0.21 6379 #quorum 3/2
6175:X 18 Sep 2024 21:39:50.451 # Next failover delay: I will not start a failover before Wed Sep 18 21:45:50 2024
6175:X 18 Sep 2024 21:39:51.853 # +config-update-from sentinel 77fc4fbc9c6c7d063d62e38f528910ab67e8d907 10.0.0.26 26379 @ mymaster 10.0.0.21 6379
6175:X 18 Sep 2024 21:39:51.853 # +switch-master mymaster 10.0.0.21 6379 10.0.0.22 6379
6175:X 18 Sep 2024 21:39:51.853 * +slave slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379
6175:X 18 Sep 2024 21:39:51.853 * +slave slave 10.0.0.21:6379 10.0.0.21 6379 @ mymaster 10.0.0.22 6379
6175:X 18 Sep 2024 21:40:09.323 * +convert-to-slave slave 10.0.0.21:6379 10.0.0.21 6379 @ mymaster 10.0.0.22 6379

    Note: Next failover delay: I will not start a failover before Wed Sep 18 21:45:50 2024 due to sentinel failover-timeout mymaster 180000

  3. Run the following command on one of the Sentinel nodes:

    $ redis-cli -p 26379 --askpass sentinel get-master-addr-by-name mymaster
Please input password: ****************
1) "10.0.0.22"
2) "6379"

    Redis node with IP 10.0.0.22 is the new Master node.

  4. Run the following command on the new Master node:

    $ redis-cli --askpass info replication
# Replication
role:master
connected_slaves:2
slave0:ip=10.0.0.23,port=6379,state=online,offset=519426,lag=1
slave1:ip=10.0.0.21,port=6379,state=online,offset=519559,lag=0
master_replid:ade635098e62dc68f3ee65acf37970f98e1a2f8a
master_replid2:8777248b814d71f4c609f437c37a529e01b7d0c5
master_repl_offset:519692
second_repl_offset:195253
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:519692

Sentinel node failure

  1. Run the following command on the first Sentinel node:

    $ sudo systemctl stop redis-sentinel

  2. Check redis-sentinel.log on the second Sentinel node:

    tail -1 /var/log/redis/redis-sentinel.log
5587:X 18 Sep 2024 22:14:36.112 # +sdown sentinel d01aa131dc0ba6cc4069f3ec67bf6b4fa6b501f8 10.0.0.24 26379 @ mymaster 10.0.0.22 6379

  3. Run the following command on Master node:

    $ redis-cli --askpass client pause 40000
Please input password: ****************
OK

  4. Check redis-sentinel.log on the second Sentinel node:

    $ tail -10 /var/log/redis/redis-sentinel.log
5587:X 18 Sep 2024 22:20:49.232 # +failover-state-reconf-slaves master mymaster 10.0.0.22 6379
5587:X 18 Sep 2024 22:20:49.255 * +slave-reconf-sent slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379
5587:X 18 Sep 2024 22:20:49.691 # -odown master mymaster 10.0.0.22 6379
5587:X 18 Sep 2024 22:20:54.812 # -sdown master mymaster 10.0.0.22 6379
5587:X 18 Sep 2024 22:20:56.253 * +slave-reconf-inprog slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379
5587:X 18 Sep 2024 22:20:56.254 * +slave-reconf-done slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379
5587:X 18 Sep 2024 22:20:56.278 # +failover-end master mymaster 10.0.0.22 6379
5587:X 18 Sep 2024 22:20:56.278 # +switch-master mymaster 10.0.0.22 6379 10.0.0.21 6379
5587:X 18 Sep 2024 22:20:56.279 * +slave slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.21 6379
5587:X 18 Sep 2024 22:20:56.279 * +slave slave 10.0.0.22:6379 10.0.0.22 6379 @ mymaster 10.0.0.21 6379

    According to the log the node with IP 10.0.0.21 is the new Master node.

  5. Run the following command on the second Sentinel node:

    $ sudo systemctl stop redis-sentinel

  6. Check redis-sentinel.log on the third Sentinel node:

    tail -1 /var/log/redis/redis-sentinel.log
5406:X 18 Sep 2024 22:26:57.116 # +sdown sentinel 4a1264d7e6946072945569a8a6c91ad5bff182aa 10.0.0.25 26379 @ mymaster 10.0.0.21 6379

    At this point, the only single Sentinel node 10.0.0.26 running, and the Redis node with IP 10.0.0.21 is the new Master node.

  7. Run the following command on Master node:

    $ redis-cli --askpass client pause 40000
Please input password: ****************
OK

  8. Check redis-sentinel.log on the third Sentinel node:

    $ tail -2 /var/log/redis/redis-sentinel.log
5406:X 18 Sep 2024 22:26:57.116 # +sdown sentinel 4a1264d7e6946072945569a8a6c91ad5bff182aa 10.0.0.25 26379 @ mymaster 10.0.0.21 6379
5406:X 18 Sep 2024 22:42:37.815 # +sdown master mymaster 10.0.0.21 6379

    According to the log the Master node is down, but there was no voting for the new leader. This is due to the missing of quorum among the Sentinel nodes.

  9. Don't forget to start redis-sentinel service on the first and second Sentinel nodes:

    $ sudo systemctl start redis-sentinel

Congrats! You have configured Redis Sentinel.

Summary

In this post, we’ve seen:

  • how to set up Redis Master-Slave replication
  • how to set up Redis Sentinel
  • hot to test the failover of the high availability Redis using Sentinel.

Top comments (0)

Read next

mdarifulhaque profile image

2461. Maximum Sum of Distinct Subarrays With Length K

MD ARIFUL HAQUE -

seif_ali profile image

C++

Seif Ali Atrous -

nausaf profile image

Aborted attempt to return meaningful Problem Details responses for model binding errors in an ASP.NET Core Minimal API

nausaf -

ivanivanovv profile image

Launching Vratix - open source Node.js API modules

Ivan Ivanov -