Redis4/5---Sentry High Availability Configuration

1. Basic Concepts

(1) The core role of the Sentinel mechanism


When there is a problem with the main server, the client will ask the sentinel;
Normally, the client connects directly to the main server.

(2) Core operation process

  • Server discovery and monitoring inspection process

    Method to realize:
#run the client
[root@localhost redis]# ./bin/redis-cli -p 26380
#The client sends the command to ask the sentinel for the master server information:
127.0.0.1:26380> SENTINEL get-master-addr-by-name mymaster
1) "192.168.11.130"
2)"6380"
#Sentinel reads configuration files and monitors redis instances
[root@localhost redis]# cat conf/sentinel-26380.conf

  • Failover Process

    Implementation method: 7 core concepts

  • ①How does the sentinel know Redis master-slave information (automatic discovery mechanism)

    In the sentry configuration file, the information of the master in the master-slave cluster is saved, and the master-slave information can be automatically discovered through the info replication command.

  • ②What is master subjective offline

    Subjective offline: a single sentinel thinks that the redis instance can no longer provide services
    Detection mechanism: Sentinel sends a ping request to redis. The three cases of +PONG, -LOADING, and -MASTERDOWN are regarded as normal, and other replies are regarded as invalid.

  • ③What is objective offline
    Objective offline: a certain number of sentinels believe that the master has been offline
    Detection mechanism: When the sentinel subjectively believes that the master is offline, it will ask other sentinels whether they think it is through the SENTINEL is-master-down-by-addr command.
    The master has been offline. If a consensus is reached (the number of quorum s is reached), the master node will be considered to be offline objectively and the failover process will begin.
    The configuration item corresponding to the configuration file: sentinel monitor mymaster 192.168.11.131 6380 2

  • ④ How to communicate between sentinels (automatic discovery between sentries)

  • ⑤ Which Sentinel is responsible for failover? (Sentry leadership election mechanism)

Based on the election mechanism implemented by the Raft algorithm, the process is briefly described as follows:

canvassing stage: each sentinel node hopes to become the leader;

After the sentinel node receives the canvassing command, if it has not received or agreed to the request of other sentinel nodes, it will agree to the sentinel's request (each sentinel only holds one consent vote);

If the sentinel node finds that it has more than half of its votes, it becomes the leader and performs failover;

After the voting is over, if the actual failover operation is not performed within the failover-timeout period, the election will be rescheduled.
Click to view the raft agreement icon

  • ⑥slave election mechanism

From the server example, filter in order:

slave node status, not S_DOWN,O_DOWN,DISCONNECTED
 Judgment rules: ( down-after-milliseconds * 10)+milliseconds_since_master_is_in_SDOWN_state
SENTINEL slaves mymaster
priority
redis.conf One of the configuration items: slave-priority The smaller the value, the higher the priority
data synchronization
Replication offset processed
the smallest run id
run id Comparison scheme: lexicographical order, ASCII code
  • ⑦ Final master-slave switching process

For the slave node that is about to become the master, withdraw it from the master-slave cluster
Automatic execution: slaveof NO ONE

For other slave nodes, make them slaves of the new master
Automatic execution: slaveof new_master_host new_master_port

2. Configure server clusters and Sentinel high-availability clusters

(1) Download and install Redis

Refer to the official website documentation

$ wget http://download.redis.io/releases/redis-6.0.5.tar.gz
$ tar xzf redis-6.0.5.tar.gz
$ cd redis-6.0.5
$ make

(2) Create a folder

mdkir /usr/local/redis/conf
mdkir /usr/local/redis/data
mdkir /usr/local/redis/logs

(3) Prepare the server configuration file

#Configuration files have been streamlined
#background start meaning
daemonize yes
#Port number (if it is started on the same server, pay attention to modify it to a different port
port 6380
#IP binding, redis is not recommended to open to the public network, you can directly bind 0.0.0.0
bind 0.0.0.0
#This file will be automatically generated (if it is started on the same server, pay attention to modifying it to a different port)
pidfile "/var/run/redis_6380.pid"
# Generated by CONFIG REWRITE
user default on nopass ~* +@all
dir "/usr/local/redis"

(4) Configure the master-slave and view the master-slave server cluster

Make a soft connection and you can use it globally after completion

#The front is the path of the redis installation, and the back is to put the redis-server under /usr/local/bin/redis
#Server
[root@localhost ~]# ln -s /root/redis-6.0.5/src/redis-server /usr/local/redis/bin/redis-server
#client
[root@localhost ~]# ln -s /root/redis-6.0.5/src/redis-cli /usr/local/redis/bin/redis-cli
  • Start three Redis
[root@localhost redis]# /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis_6380.conf
[root@localhost redis]# /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis_6381.conf
[root@localhost redis]# /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis_6382.conf
  • Configured as 1 master and 2 slaves
[root@localhost redis]#/usr/local/redis/bin/redis-cli -p  6381 slaveof 192.168.11.131 6380
[root@localhost redis]#/usr/local/redis/bin/redis-cli -p  6382 slaveof 192.168.11.131 6380

  • Check the cluster
[root@localhost redis]# /usr/local/redis/bin/redis-cli -p  6380 info Replication

(5) Prepare the sentinel configuration file

#Configuration file: sentinel.conf, which will be dynamically modified during sentinel running
#If sentinel is restarted, it will reply to the status of the previously monitored redis cluster according to this configuration
#Bind Ip
bind 0.0.0.0
#Background process
daemonize yes
#The default is yes, if the specified password or specified IP is set, the external network cannot be accessed.
protected-mode no
#Sentinel port, the client discovers redis through this port
port 26380
#The sentinel specifies its own IP, which can also be automatically discovered by manual settings and used to communicate with other sentries
#sentinel announce-ip
#temporary folder
dir "/tmp"
#log
logfile "/usr/local/redis/logs/sentinel-26380.log"
#The name of the master monitored by sentinel is mymaster, the initial address is 192.168.11.131 6380, 2 represents two and two
#The death of the above sentinel mission is considered the real death
sentinel monitor mymaster 192.168.11.131 6380 2
#Send a heartbeat PING to check if the master is alive
#If the master does not respond to PING within a certain time frame, or replies with an error message, then the sentinel will subjectively (unilaterally) think that the master is unavailable
sentinel down-after-milliseconds mymaster 1000
#If the failover operation cannot be completed within this time (ms), the failover event will be executed
sentinel failover-timeout mymaster 3000
#Specifies that when performing a failover, there can be at most multiple slave Redsi instances synchronizing the new master instance, and at the slave Rdis instance
#In the case of many instances, the smaller the number, the longer the synchronization time, and the longer the time required to complete the failover.
sentinel parallel-syncs mymaster 1

(6) Start the sentinel cluster

/usr/local/redis/bin/redis-server /usr/local/redis/conf/sentinel-26380.conf --sentinel
/usr/local/redis/bin/redis-server /usr/local/redis/conf/sentinel-26381.conf --sentinel
/usr/local/redis/bin/redis-server /usr/local/redis/conf/sentinel-26382.conf --sentinel


At this point, the server and sentinel cluster are built.

3. Test the high availability performance of the Sentinel cluster

test code

  • Sentinel cluster code
@Configuration
@Profile("sentinel")
class SentinelRedisAppConfig {
    @Bean
    public LettuceConnectionFactory redisConnectionFactory() {
        System.out.println("Use the sentinel version");
        RedisSentinelConfiguration sentinelConfig = new RedisSentinelConfiguration()
                .master("mymaster")
                // Sentinel address
                .sentinel("192.168.11.131", 26380)
                .sentinel("192.168.11.131", 26381)
                .sentinel("192.168.11.131", 26382);
        return new LettuceConnectionFactory(sentinelConfig);
    }
}
  • Sentinel cluster unit test
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration("classpath:applicationContext.xml")
@ActiveProfiles("sentinel") // set profile
public class SentinelTests {

    @Autowired
    StringRedisTemplate stringRedisTemplate;

    @Test
    public void test() throws InterruptedException {
        // Every second, operate redis to see the final effect
        int i = 0;
        while (true) {
            i++;
            stringRedisTemplate.opsForValue().set("test-value", String.valueOf(i));
            System.out.println("Revise test-value value is: " + i);
            Thread.sleep(1000L);
        }
    }

}

Test 1: Stop the main service

Is the program working properly: Yes.


Test 2: Stop the Sentinel

Is the program working properly: Yes.
At this time, redis and the client are connected normally, and the client does not need a sentinel, so stopping the sentinel does not affect the program.

Test 3: Stop the main server + sentinel

Is the program working properly: Yes.

The client went to the sentinel in the order of the sentinels configured in the program, and found that the sentinel died.
In a short time, a warning message appeared, but after many retries by the client, it went to find the next sentinel, and finally the program continued to run normally.

  • redis clusters are deployed in production environments, usually on different servers, different cabinets, or even different computer rooms. Try to ensure that the server does not hang due to hardware failure.
  • Sentinels only work when the client does not know which master server to connect to.

4. Sentinel election process

Sentinel election

log file during sentry election

Monitor other servers: View the changes of other servers after the sentinel election changes the master server

Posted by ababmxking on Sun, 29 May 2022 22:51:21 +0530