Redis closing connections to clients while saving changed keys in background

Question

I have one Redis server on EC2 with 2 app servers connected to it. All small/medium. Traffic is not high; only 10 changed keys in 300 seconds. I started noticing on the app servers connection errors to the Redis machine:

 Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

At first I thought it was a problem with my pool configuration or the java client I am using to interface with Redis, however I quickly debunked this theory when I noticed that both app servers would always generate these exceptions at the exact same time and they'd always come in bunches. Then I looked at redis.log and noticed the following output while the errors would appear:

[13939] 08 May 22:31:05.051 * 10 changes in 300 seconds. Saving... [13939] 08 May 22:31:05.342 * Background saving started by pid 13945 [13939] 08 May 22:31:09.357 - DB 0: 606477 keys (0 volatile) in 1048576 slots HT. [13939] 08 May 22:31:09.357 - 3 clients connected (0 slaves), 764180208 bytes in use [13939] 08 May 22:31:14.542 - DB 0: 606477 keys (0 volatile) in 1048576 slots HT. [13939] 08 May 22:31:25.947 - 3 clients connected (0 slaves), 764180208 bytes in use [13939] 08 May 22:31:25.947 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.947 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.947 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.947 - Accepted 10.123.29.90:56301 [13939] 08 May 22:31:25.947 - Accepted 10.42.105.60:35315 [13939] 08 May 22:31:25.947 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.947 - Accepted 10.123.29.90:56302 [13939] 08 May 22:31:25.947 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.947 - Accepted 10.42.105.60:35317 [13939] 08 May 22:31:25.947 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.947 - Accepted 10.123.29.90:56306 [13939] 08 May 22:31:25.948 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.948 - Accepted 10.42.105.60:35318 [13939] 08 May 22:31:25.948 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.948 - Accepted 10.123.29.90:56308 [13939] 08 May 22:31:25.948 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.948 - Accepted 10.42.105.60:35319 [13939] 08 May 22:31:25.948 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.948 - Accepted 10.42.105.60:35320 [13939] 08 May 22:31:25.948 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.948 - Accepted 10.123.29.90:56310 [13939] 08 May 22:31:25.948 - Error writing to client: Connection reset by peer [13939] 08 May 22:31:25.948 - Accepted 10.42.105.60:35322 [13939] 08 May 22:31:27.652 - Accepted 10.42.105.60:35327 [13939] 08 May 22:31:27.872 - Accepted 10.42.105.60:35329 [13945] 08 May 22:31:27.926 * DB saved on disk

The errors only occur while Redis is saving new data in the background. I am using Redis 2.6. Any help is appreciated.

EDIT: Redis connection pool configuration below using spring-data

<bean id="redisPoolConfig" class="redis.clients.jedis.JedisPoolConfig" lazy-init="false"
      p:maxTotal="500"
      p:maxIdle="20"
      p:testOnBorrow="true"
      p:testOnCreate="true"
      p:testOnReturn="true"
      p:maxWaitMillis="30000"
    />
<bean id="jedisConnectionFactory" class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory"
      p:hostName="${REDIS_HOST}"
      p:port="${REDIS_PORT}"
      p:usePool="true"
      p:poolConfig-ref="redisPoolConfig"
    />

Answer 1

I think I found the answer. After upgrading to 2.8.9 and restarting, the log alerted me to my OS limiting the number of open file descriptors to 1024, which can easily be exceeded with a running Redis instance. So what probably was happening was the background save process was opening up new file descriptors, which prevented new connections from being accepted since each new client connection opens a file. After increasing the limit to 10000 via ulimit -n everything seems to be functioning correctly.