JedisPool resource pool optimization

Reasonable JedisPool resource pool parameter settings can effectively improve Redis performance. This document will describe in detail the use of JedisPool and the parameters of resource pool, and provide suggestions for optimizing configuration.

usage method

Take Jedis 2.9.0 as an example, its Maven dependencies are as follows:

<dependency>
    <groupId>redis.clients</groupId>
    <artifactId>jedis</artifactId>
    <version>2.9.0</version>
    <scope>compile</scope>
</dependency>

Jedis uses Apache Commons-pool2 to manage the resource pool. When defining JedisPool, pay attention to its key parameter GenericObjectPoolConfig (resource pool). An example of using this parameter is as follows. Please refer to the following for the description of the parameters.

GenericObjectPoolConfig jedisPoolConfig = new GenericObjectPoolConfig();
jedisPoolConfig.setMaxTotal(...);
jedisPoolConfig.setMaxIdle(...);
jedisPoolConfig.setMinIdle(...);
jedisPoolConfig.setMaxWaitMillis(...);
...

The initialization method of JedisPool is as follows:

// redisHost is the instance IP, redisPort is the instance port, and redisPassword is the instance password. Timeout is both a connection timeout and a read-write timeout
JedisPool jedisPool = new JedisPool(jedisPoolConfig, redisHost, redisPort, timeout, redisPassword//);
//Execute the command as follows
Jedis jedis = null;
try {
    jedis = jedisPool.getResource();
    //Specific commands
    jedis.executeCommand()
} catch (Exception e) {
    logger.error(e.getMessage(), e);
} finally {
    //In JedisPool mode, Jedis will be returned to the resource pool
    if (jedis != null) 
        jedis.close();
}

Parameter description

Jedis connection is the resource managed by JedisPool in the connection pool. JedisPool ensures that the resources are within a controllable range and thread safety. Using reasonable GenericObjectPoolConfig configuration can improve Redis service performance and reduce resource overhead. The following two tables will describe some important parameters and provide setting suggestions.

parameter	explain	Default value	proposal
maxTotal	Maximum number of connections in the resource pool	8	See Key parameter setting suggestions.
maxIdle	Maximum number of free connections allowed for the resource pool	8	See Key parameter setting suggestions.
minIdle	Minimum number of free connections guaranteed by the resource pool	0	See Key parameter setting suggestions.
blockWhenExhausted	When the resource pool is exhausted, does the caller wait? The following maxWaitMillis will take effect only when the value is true.	true	Default values are recommended.
maxWaitMillis	When the resource pool connection is exhausted, the maximum waiting time of the caller is in milliseconds.	-1 (never timeout)	Default values are not recommended.
testOnBorrow	Whether to ping the connection validity when borrowing a connection from the resource pool. Invalid connections detected will be removed.	false	When the traffic is large, it is recommended to set it to false to reduce the overhead of a ping.
testOnReturn	Whether to ping the connection validity when returning the connection to the resource pool. Invalid connections detected will be removed.	false	When the traffic is large, it is recommended to set it to false to reduce the overhead of a ping.
jmxEnabled	Enable JMX monitoring	true	It is recommended to start. Please note that the application itself also needs to be started.

Idle Jedis object detection is completed by a combination of the following four parameters.

name	explain	Default value	proposal
testWhileIdle	Whether to monitor the validity of the connection through the ping command during idle resource monitoring. Invalid connections will be destroyed.	false	true
timeBetweenEvictionRunsMillis	Detection cycle of idle resources (in milliseconds)	-1 (not tested)	It is recommended to set the cycle. You can choose the cycle by yourself. You can also use the configuration in JedisPoolConfig below by default.
minEvictableIdleTimeMillis	The minimum idle time (in milliseconds) of resources in the resource pool. When this value is reached, idle resources will be removed.	1800000 (i.e. 30 minutes)	It can be determined according to its own business. It is generally the default value. You can also consider using the configuration in JeidsPoolConfig below.
numTestsPerEvictionRun	The number of resources detected each time when idle resources are detected.	3	Fine tuning can be performed according to the number of application connections. If it is set to - 1, idle monitoring will be performed on all connections.

For ease of use, Jedis provides JedisPoolConfig, which inherits some settings of GenericObjectPoolConfig on idle detection.

public class JedisPoolConfig extends GenericObjectPoolConfig {
  public JedisPoolConfig() {
    // defaults to make your life with connection pool easier :)
    setTestWhileIdle(true);
    //
    setMinEvictableIdleTimeMillis(60000);
    //
    setTimeBetweenEvictionRunsMillis(30000);
    setNumTestsPerEvictionRun(-1);
    }
}

Note you can view all default values in org.apache.commons.pool2.impl.BaseObjectPoolConfig.

Key parameter setting suggestions

maxTotal (maximum connections)

There are many factors to consider when setting maxTotal reasonably, such as:

• Redis concurrency expected by the business;
• Client execution command time;
• Redis resources, such as nodes (such as the number of ECS applications) * maxTotal, cannot exceed the maximum number of redis connections (which can be viewed on the instance details page);
• Resource overhead, for example, although you want to control idle connections, you don't want unnecessary overhead caused by frequent release and creation of connections in the connection pool.

Assuming that the average time taken for a command, that is, the border return resource plus Jedis to execute commands (including network time), is about 1ms, the QPS of a connection is about 1s/1ms = 1000, and the QPS of a single Redis expected by the business is 50000 (the total number of QPS/Redis fragments of the business), then the required resource pool size (i.e. MaxTotal) is 50000 / 1000 = 50 in theory.

But in fact, this is only a theoretical value. In addition, some resources should be reserved, so maxTotal can be larger than the theoretical value. The larger the value, the better. On the one hand, too many connections will occupy the resources of the client and server. On the other hand, for Redis, a high QPS server, if there is a blocking of large commands, even setting a large resource pool will not help.

maxIdle and minIdle

In fact, the maxIdle is the maximum number of connections required by the business. maxTotal is to give the margin, so the maxIdle should not be set too small, otherwise there will be new Jedis (new connection) overhead, and the minIdle is to control the detection of free resources.

The best performance of connection pool is maxTotal=maxIdle, which avoids the performance interference caused by connection pool scaling. If your business has sudden peak access, it is recommended to set the values of these two parameters to be equal; If the concurrency is small or the maxIdle is set too high, it will lead to unnecessary waste of connection resources.

You can evaluate the size of the connection pool used by each node as a whole according to the actual total QPS and the size of the client calling Redis.

Use monitoring to obtain reasonable values

In the actual environment, the more reliable method is to try to obtain the best value of parameters through monitoring. It can be considered to realize monitoring through JMX and other methods, so as to find reasonable values.

common problem

Insufficient resources

The following two cases are that the resource cannot be obtained from the resource pool.

• Timeout:

  redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
  ...
  Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
  at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449)
  

blockWhenExhausted is false, so it will not wait for the resource to be released:

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
...
Caused by: java.util.NoSuchElementException: Pool exhausted
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:464)

The reason for such exceptions is not necessarily that the resource pool is not large enough. See** Key parameter setting suggestions **Analysis in. It is recommended to check the network, resource pool parameter settings, resource pool monitoring (if JMX is monitored), code (for example, jedis.close() is not executed), slow query, DNS, etc.

Preheat JedisPool

For some reasons (such as setting a small timeout), the project may timeout after successful startup. When JedisPool defines the maximum number of resources and the minimum number of free resources, Jedis connections will not be created in the connection pool. When the pool is used for the first time, if there are no resources in the pool, new Jedis will be used first, and then put into the resource pool. This process will have a certain time overhead. Therefore, it is recommended to preheat the JedisPool based on the minimum idle quantity after defining the JedisPool. The examples are as follows:

List<Jedis> minIdleJedisList = new ArrayList<Jedis>(jedisPoolConfig.getMinIdle());
 
for (int i = 0; i < jedisPoolConfig.getMinIdle(); i++) {
    Jedis jedis = null;
    try {
        jedis = pool.getResource();
        minIdleJedisList.add(jedis);
        jedis.ping();
    } catch (Exception e) {
        logger.error(e.getMessage(), e);
    } finally {
    }
}
 
for (int i = 0; i < jedisPoolConfig.getMinIdle(); i++) {
    Jedis jedis = null;
    try {
        jedis = minIdleJedisList.get(i);
        jedis.close();
    } catch (Exception e) {
        logger.error(e.getMessage(), e);
    } finally {
    
    }
}