Deploy HA Redis Cluster with Sentinel on Rocky Linux 8|AlmaLinux 8

Posted on 162 views

Redis is a free and open-source, in-memory data store used by many developers around the world as a database, cache, message broker, and streaming engine. It gains massive popularity with its broad support, high availability, performance, and flexibility. Redis supports several data structures such as lists, hashes, sets, sorted sorts, strings, streams e.t.c. The capabilities provided by Redis are well explained below:

  • Extensibility: It has a module API that can be used to build extensions in C, C++, and Rust languages.
  • Persistence: It keeps that dataset in memory to allow fast access. It can also persist the data by writing it to a permanent storage disk to survive reboots.
  • High availability: It offers replication with automatic failover for standalone and clustered modes.
  • Programmability: It supports server-side scripting with Lua and server-side stored procedures with Redis Functions.

Redis replication is the process of creating exact copies of the Redis master instance. The replication is asynchronous by default. The master continues to handle queries while the replicas perform the initial synchronization or a partial resynchronization. Replication is mainly done to improve read throughput and circumvent data loss in cases of node failure.

The Redis replication can be managed manually using the SLAVEOF or REPLICAOF.To make this easier, Redis Sentinel is used. This is a process that is used to automate and monitor the Redis replication failover and switchover. Sentinel requires a number of instances and then works as a group of decision-makers to determine which node is up and which one is down before triggering a failover.

The below illustration will help you understand how Sentinel works.

Deploy-HA-Redis-Cluster-with-Sentinel-on-Rocky-Linux-AlmaLinux-1

Aside from high availability, the Redis Sentinel offers more other features that include:

  • Monitoring: It constantly checks if both the master and replica instances are running as required.
  • Automatic failover: In case the master is not working as required, Sentinel triggers a failover and the replica is promoted to a master. The other replicas attached are then configured to use this new master.
  • Configuration provider: It provides an authoritative source for clients’ service discovery. The clients connect to Sentinels to obtain the address of the current Redis master.
  • Notification: It can be used to notify the system admin when something is wrong with any of the monitored Redis instances.

This guide offers a deep demonstration of how to deploy HA Redis Cluster with Sentinel on Rocky Linux 8 | AlmaLinux 8.

Setup Requirements.

It is recommended to have at least 3 Sentinel instances for a sturdy deployment. This guide will work with 3 servers as below:

HOSTNAME TASK IP ADDRESS
Redis_master Master 192.168.205.2
Redis_replica1 slave1 192.168.205.3
Redis_replica2 slave2 192.168.205.33

Step 1 – Install Redis on Rocky Linux 8 | AlmaLinux 8

Redis is available in the default Rocky Linux 8 | AlmaLinux 8 repositories and can be installed on all the 3 nodes with the command:

sudo dnf install vim @redis -y

Once installed, start and enable the service on all the 3 nodes as shown:

sudo systemctl start redis
sudo systemctl enable redis

Verify the status of the server:

$ systemctl status redis
● redis.service - Redis persistent key-value database
   Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/redis.service.d
           └─limit.conf
   Active: active (running) since Thu 2022-06-23 05:06:50 EDT; 14s ago
 Main PID: 2784 (redis-server)
    Tasks: 4 (limit: 23682)
   Memory: 6.6M
   CGroup: /system.slice/redis.service
           └─2784 /usr/bin/redis-server 127.0.0.1:6379

Step 2 – Configure Redis Replication On Rocky Linux 8 | AlmaLinux 8

Normally, Redis can be configured using the /etc/redis.conf file. We will configure the master and slave instances as shown.

A. Configure Redis Master server

Create a backup file and open the configuration file for editing:

sudo cp /etc/redis.conf  /etc/redis.conf.bak
sudo vim /etc/redis.conf

Begin by configuring the bind-address:

bind 127.0.0.1 192.168.205.2

Next set the protected-mode to no, to allow communication with the slaves.

protected-mode no

You can secure the master instance using the requirepassdirective:

requirepass  StrongPassword

Set the service to interact with systemd.

supervised systemd

Save the configuration file and restart the service

sudo systemctl daemon-reload
sudo systemctl restart redis

Allow the service through the firewall:

sudo firewall-cmd --zone=public --permanent --add-port=6379/tcp
sudo firewall-cmd --reload

B. Configure Redis Replicas

We will then configure our two replicas as below:

sudo cp /etc/redis.conf /etc/redis.conf.bak
sudo vim /etc/redis.conf

For the replicas, make the below changes, remember to replace values where required.

bind 127.0.0.1 192.168.205.3
protected-mode no
supervised systemd
replicaof 192.168.205.2 6379
masterauth  StrongPassword

Replace the bind IP address with the appropriate IP address of the replica. The replicaofdirective is used to provide replication of the master server.

Save the file and restart the services:

sudo systemctl daemon-reload
sudo systemctl restart redis

Allow the service through the firewall:

sudo firewall-cmd --zone=public --permanent --add-port=6379/tcp
sudo firewall-cmd --reload

C. Verify the Replication

Once the above configurations have been made, verify the replication:

  • On the Master node:
$ redis-cli 
127.0.0.1:6379> AUTH StrongPassword
127.0.0.1:6379> info replication

Sample Output:

Deploy-HA-Redis-Cluster-with-Sentinel-on-Rocky-Linux-AlmaLinux-1-1

  • On the Replicas:
redis-cli info replication

Sample Output.

Deploy-HA-Redis-Cluster-with-Sentinel-on-Rocky-Linux-AlmaLinux-2

From the above output, we can all agree that we have configured Redis replication on Rocky Linux 8 | AlmaLinux 8

Step 3 – Configure Redis Sentinel on Rocky Linux 8 | AlmaLinux 8

The Redis sentinel was installed when installing the Redis server. Start and enable the service on all the 3 nodes as shown:

sudo systemctl start redis-sentinel
sudo systemctl enable redis-sentinel

Check the status of the service:

$ systemctl status redis-sentinel
● redis-sentinel.service - Redis Sentinel
   Loaded: loaded (/usr/lib/systemd/system/redis-sentinel.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/redis-sentinel.service.d
           └─limit.conf
   Active: active (running) since Thu 2022-06-23 05:27:28 EDT; 25s ago
 Main PID: 3127 (redis-sentinel)
    Tasks: 4 (limit: 23682)
   Memory: 6.6M
   CGroup: /system.slice/redis-sentinel.service
           └─3127 /usr/bin/redis-sentinel *:26379 [sentinel]

To configure the Redis Sentinel service we will edit the file below on all the 3 nodes:

sudo cp /etc/redis-sentinel.conf /etc/redis-sentinel.conf.bak
sudo vim /etc/redis-sentinel.conf

First, ensure the service listens on the desired port, the default port is 26379

port 26379

For Sentinel to monitor our master server, we need to configure it on the nodes as shown:

#On Master Server(Sentinel1)/Replica1(Sentinel2)/Replica2(Sentinel3)
.......
sentinel monitor mymaster 192.168.205.2 6379 2
sentinel auth-pass mymaster StrongPassword
sentinel down-after-milliseconds mymaster 10000
sentinel parallel-syncs mymaster 1

In the file:

  • mymaster can be replaced with a desired name and password.
  • Replace 192.168.205.2 with the master IP address
  • Remember to put the sentinel monitor line before the sentinel auth-passline. Otherwise, the service will fail to restart.
  • The sentinel down-after-millisecondsis used to set the time required for the master to be considered as failed if no response is received.
  • parallel-syncsset the number of replicas to be promoted to the new master after a failover.

Save the files and restart the services:

sudo systemctl restart redis-sentinel

Allow the service through the firewall on all the nodes:

sudo firewall-cmd --zone=public --permanent --add-port=26379/tcp
sudo firewall-cmd --reload

Step 4 – Check the Redis Sentinel Setup Status

Now we can check the status of the Redis Sentinel Setup:

  • On the master node:
redis-cli -p 26379 info sentinel

Execution output:

Deploy-HA-Redis-Cluster-with-Sentinel-on-Rocky-Linux-AlmaLinux-3

You can view detailed information with the command:

redis-cli -p 26379 sentinel master mymaster

Execution output:

Deploy-HA-Redis-Cluster-with-Sentinel-on-Rocky-Linux-AlmaLinux-4

To check the status of the slaves, use the command:

redis-cli -p 26379 sentinel slaves mymaster

Step 5 – Redis Sentinel Failover Testing

We will test the automatic failover feature provided by Sentinel. To do this, stop the Redis service on the master server.

sudo systemctl stop redis

After the 10 seconds, (the down-after-milliseconds set) Check the Sentinel log file:

sudo cat /var/log/redis/sentinel.log

Sample Output:

Deploy-HA-Redis-Cluster-with-Sentinel-on-Rocky-Linux-AlmaLinux-5-1024x321

From the output, we can see that our replica 192.168.205.3 has been promoted to the new master.

Get detailed info from the Redis CLI

redis-cli -h 192.168.205.3 -p 26379 sentinel masters

Sample Output:

Deploy-HA-Redis-Cluster-with-Sentinel-on-Rocky-Linux-AlmaLinux-6

That was enough learning!

That marks the end of this guide on how to deploy HA Redis Cluster with Sentinel on Rocky Linux 8 | AlmaLinux 8. I hope this was of importance to you.

coffee

Gravatar Image
A systems engineer with excellent skills in systems administration, cloud computing, systems deployment, virtualization, containers, and a certified ethical hacker.