Jedis&Redis persistence

Jedis

Programming language and redis

⚫ Jedis is used to connect redis services in Java language and provide corresponding operation API s
⚫ Java language connection redis service

  • Jedis
  • SpringData Redis
  • Lettuce

⚫ C ,C++ ,C# ,Erlang,Lua ,Objective-C ,Perl ,PHP ,Python ,Ruby ,Scala

Jedis HelloWorld

Preparation

⚫ jar package import

⚫ Based on maven

<dependency> 
  <groupId>redis.clients</groupId>
  <artifactId>jedis</artifactId> 
  <version>2.9.0</version>
</dependency>

Client connection redis

⚫ Connect to redis

Jedis jedis = new Jedis("localhost", 6379);

⚫ Operation redis

jedis.set("name", "itheima");
jedis.get("name");

⚫ Close redis connection

jedis.close();

⚫ API documentation: http://xetorthio.github.io/jedis/

Jedis simple tool class development

Get connection based on connection pool

⚫ JedisPool: connection pool technology provided by Jedis

  • poolConfig: connection pool configuration object
  • host:redis service address
  • port:redis service port number
public JedisPool(GenericObjectPoolConfig poolConfig, String host, int port) {
     this(poolConfig, host, port, 2000, (String)null, 0, (String)null);
}

Package connection parameters

⚫ jedis.properties

jedis.host=192.168.40.130
jedis.port=6379
jedis.maxTotal=50
jedis.maxIdle=10

Load configuration information

⚫ Static code block initialization resource

static{
//Read configuration file to get parameter value
ResourceBundle rb = ResourceBundle.getBundle("jedis");
host = rb.getString("jedis.host");
port = Integer.parseInt(rb.getString("jedis.port"));
maxTotal = Integer.parseInt(rb.getString("jedis.maxTotal"));
maxIdle = Integer.parseInt(rb.getString("jedis.maxIdle"));
poolConfig = new JedisPoolConfig();
poolConfig.setMaxTotal(maxTotal);
poolConfig.setMaxIdle(maxIdle);
jedisPool = new JedisPool(poolConfig,host,port);
}

Get connection

⚫ The external access interface provides jedis connection objects, and the connections are obtained from the connection pool

public static Jedis getJedis(){
   Jedis jedis = jedisPool.getResource();
   return jedis;
}

Jedis link problem

  • Protection mode solution
    • ./redis-server --protected-mode no
    • Modify the bind in the configuration file to 0.0.0.0

Jedis distributed queue implementation

  1. Create a queue to save tasks. This queue can be accessed on any program (thread), any process, and any computer
  2. Assumption: you access redis to store data independently
  3. Definition: customize a class to encapsulate all added tasks and get the details of the person
    • Define an interface (store data, get data)
    • Define jedis links

Code implementation:

  • interface
package com.itheima;

import java.util.List;

public interface DistributedInterface {
    //send data
    void faSong(String key);

    //Read data
    List<String> duQu(Integer start, Integer stop);
}

  • Implementation class
package com.itheima;

import redis.clients.jedis.Jedis;

import java.util.List;

public class DistributedImpl implements DistributedInterface{
    Jedis jedis;
    private final String name = "names";
    //Encapsulated connection
    public DistributedImpl() {
         jedis = new Jedis("127.0.0.1",6379);
    }

    @Override
    public void faSong(String key) {
        jedis.lpush(name,key);
    }

    @Override
    public List<String> duQu(Integer start, Integer stop) {
        return jedis.lrange(name,start,stop);
    }
}

  • Test class
package com.itheima;

import org.junit.Test;

public class TestDistributde {
   @Test
   public void faSong(){
       DistributedInterface distributedInterface = new DistributedImpl();
       distributedInterface.faSong("zhangsan");
   }


    @Test
    public void duQu(){
        DistributedInterface distributedInterface = new DistributedImpl();
        System.out.println(distributedInterface.duQu(0,-1));
    }
}

Redis persistence

Why persist?

-Unexpected power failure


What is persistence

The working mechanism of using permanent storage media to save data and recover the saved data at a specific time is called persistence
Persistence is used to prevent accidental data loss and ensure data security

What does the persistence process hold

⚫ Save the current data status in the form of snapshot, store the data results in a simple format, and focus on the data
⚫ Save the operation process of data in the form of log, and store the operation process. The storage format is complex, and the focus is on the operation process of data

Persistence mode

◆ RDB
◆ AOF

RDB

RDB startup mode - save instruction

⚫ Manually perform a save operation

save

RDB startup mode - save instruction related configuration

⚫ Set the local database file name. The default value is dump rdb, usually set to dump port number rdb

dbfilename filename

⚫ Set storage The path of the rdb file is usually set to a directory with a large storage space. The directory name is data

dir path

⚫ Set whether to compress data when storing to the local database. The default is yes and set to no. this saves CPU running time, but the storage file becomes larger

rdbcompression yes|no

⚫ Set whether RDB format verification is performed during file reading and writing. The default value is yes and set to no. this saves 10% of the time for reading and writing. There is a risk of data corruption

rdbchecksum yes|no

RDB startup mode - working principle of save instruction



RDB startup mode - bgsave instruction

⚫ Start the background save operation manually, but not immediately

bgsave

RDB startup mode - bgsave instruction related configuration

⚫ Whether to stop saving if an error occurs in the background stored procedure. The default is yes

stop-writes-on-bgsave-error yes|no

⚫ other

dbfilename filename
dir path
rdbcompression yes|no
rdbchecksum yes|no

RDB startup mode - working principle of bgsave instruction

RDB startup mode - save configuration

⚫ Set the conditions for automatic persistence. When the number of key s changes within the limited time range reaches the specified number, persistence will be performed

save second changes

⚫ parameter
second: monitoring time range
changes: monitors the change of key
⚫ example

save 900 1
save 300 10
save 60 10000

RDB startup mode - working principle of save configuration

Comparison of three RDB startup modes

RDB special startup mode

⚫ Restart during server operation

debug reload

⚫ Specify save data when closing the server

shutdown save

⚫ Full replication

RDB benefits

⚫ RDB is a compact compressed binary file with high storage efficiency
⚫ RDB stores data snapshots of redis at a certain point in time, which is very suitable for data backup, full replication and other scenarios
⚫ RDB recovers data much faster than AOF
⚫ Application: bgsave backup is performed every X hours in the server, and RDB files are copied to the remote machine for disaster recovery.

RDB disadvantages

⚫ RDB mode, whether executing instructions or using configuration, cannot achieve real-time persistence, and has a greater possibility of data loss
⚫ Each time the bgsave instruction runs, it needs to perform a fork operation to create a child process, which will sacrifice some performance
⚫ The RDB file format is not unified among many versions of Redis, which may lead to incompatibility of data formats between different versions of services

AOF

Disadvantages of RDB storage

⚫ The storage data volume is large and the efficiency is low. Based on the snapshot idea, every read and write is all data. When the data volume is huge, the efficiency is very low
⚫ Low IO performance under large amount of data
⚫ Creating subprocesses based on fork causes additional memory consumption
⚫ Risk of data loss due to downtime

Solutions

⚫ Do not write all data, only record some data
⚫ Reduce the difficulty of distinguishing whether the data is changed, and change the recorded data to record the operation process
⚫ Record all operations to eliminate the risk of data loss

AOF concept

⚫ AOF(append only file) persistence: record each write command in an independent log, and re execute the commands in the AOF file when restarting
Achieve the purpose of data recovery. Compared with RDB, it can be simply understood as the change from recorded data to recorded data
⚫ The main function of AOF is to solve the real-time problem of data persistence. At present, AOF has become the mainstream method of Redis persistence

AOF data writing process

Start AOF related configuration

⚫ Enable the AOF persistence function. By default, it is no, that is, it is not enabled

appendonly yes|no

⚫ aof persistent file name. The default file name is appendonly aof. It is recommended to configure it as appendonly port number aof

appendfilename filename

⚫ Save path of AOF persistent file, which is consistent with RDB persistent file

dir

⚫ AOF write data policy, default to everysec

appendfsync always|everysec|no

Three strategies for AOF data writing (appendfsync)

⚫ Always: each write operation is synchronized to the data in the AOF file with zero error. It has low performance and is not recommended.

⚫ everysec (per second): synchronizes the instructions in the buffer to the AOF file every second. If the system suddenly goes down, the data within 1 second will be lost. The data accuracy is high and the performance is high. It is recommended to use it. It is also the default configuration

⚫ no (system control): the operating system controls the cycle of each synchronization to the AOF file. The overall process is uncontrollable

Problems encountered in AOF data writing

AOF override

As the commands continue to write to the AOF, the file will become larger and larger. To solve this problem, Redis has introduced the AOF rewrite mechanism to compress the file volume. Aof file redo
Write is the process of converting data in the Redis process into write commands and synchronizing them to a new AOF file. To put it simply, several commands of the same data will be executed
The result is converted into the instruction corresponding to the final result data for recording.

AOF override

⚫ Reduce disk usage and improve disk utilization
⚫ Improve persistence efficiency, reduce persistent write time, and improve IO performance
⚫ Reduce data recovery time and improve data recovery efficiency

AOF rewrite rule

⚫ Data with timeliness in the process, and the data has timed out, will no longer be written to the file
⚫ Invalid instructions for non write classes will be ignored and only write commands for final data will be retained
For example, del key1, hdel key2, srem key3, set key4111, set key4222, etc. Although the select instruction does not change the data, it changes the storage location of the data. Such commands also need to be recorded
⚫ Merge multiple write commands for the same data into one command
For example, lpush list1 a, lpush list1 b, and lpush list1 c can be converted to: lpush list1 a b c.
To prevent client-side buffer overflow caused by excessive data volume, write up to 64 elements per instruction for list, set, hash, zset and other types

AOF override mode

⚫ Manual override

bgrewriteaof

⚫ Auto override

auto-aof-rewrite-min-size size
auto-aof-rewrite-percentage percentage

AOF manual rewrite -- how the bgrewriteaof instruction works

AOF auto override mode

⚫ Auto override trigger condition settings

auto-aof-rewrite-min-size size
auto-aof-rewrite-percentage percent

⚫ Automatically rewrite trigger comparison parameters (run the instruction info Persistence to obtain specific information)

aof_current_size
aof_base_size

⚫ Auto override trigger condition

AOF workflow

AOF rewrite process


Differences between RDB and AOF

RDB VS AOF

Confusion about the choice of RDB and AOF

⚫ It is very sensitive to data. It is recommended to use the default AOF persistence scheme

  • The AOF persistence policy uses everysecond and fsync every second. With this strategy, redis can still maintain good processing performance. When
    In case of an existing problem, data within 0-1 seconds will be lost at most.

  • Note: due to the large storage volume of AOF files and the slow recovery speed

⚫ For the validity of data presentation phase, it is recommended to use RDB persistence scheme

  • The data can be well maintained without loss in the phase (the phase is manually maintained by the developer or the operation and maintenance personnel), and the recovery speed is fast. The phase
    Point data recovery usually adopts RDB scheme
  • Note: using RDB to realize compact data persistence will reduce Redis's performance. A careful summary:

⚫ Comprehensive comparison

  • The choice between RDB and AOF is actually a trade-off. Each has its advantages and disadvantages
  • If you can't afford data loss within a few minutes and are very sensitive to business data, select AOF
  • If it can withstand data loss within a few minutes and pursues the recovery speed of large data sets, RDB is selected
  • RDB for disaster recovery
  • The dual insurance policy enables both RDB and AOF. After restart, Redis gives priority to using AOF to recover data and reduce the amount of lost data

Tags: Database Redis

Posted by fredroines on Tue, 31 May 2022 07:16:14 +0530