redis persistence problem handling

This is a recent problem encountered in development because redis is frequently used as an intermediate query operation, suddenly fails, the service aborts, and an exception is thrown

2020-01-09 09:34:17.848 ERROR 25703 --- [ XNIO-2 task-83] o.z.p.spring.web.advice.AdviceTrait      : Internal Server Error

org.springframework.data.redis.RedisSystemException: Error in execution; nested exception is io.lettuce.core.RedisCommandExecutionException: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:54)
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:52)
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:257)
    at org.springframework.data.redis.connection.lettuce.LettuceHashCommands.convertLettuceAccessException(LettuceHashCommands.java:445)
    at org.springframework.data.redis.connection.lettuce.LettuceHashCommands.hSet(LettuceHashCommands.java:70)
    at org.springframework.data.redis.connection.DefaultedRedisConnection.hSet(DefaultedRedisConnection.java:827)
    at org.springframework.data.redis.connection.DefaultStringRedisConnection.hSet(DefaultStringRedisConnection.java:501)
    at org.springframework.data.redis.core.DefaultHashOperations.lambda$put$8(DefaultHashOperations.java:178)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184)
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:95)
    at org.springframework.data.redis.core.DefaultHashOperations.put(DefaultHashOperations.java:177)
    at com.smpteam.aaaa.service.impl.RedisServiceImpl.setESSession(RedisServiceImpl.java:59)
    at com.smpteam.aaaa.service.impl.RedisServiceImpl$$FastClassBySpringCGLIB$$11b51629.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    . . .
Caused by: io.lettuce.core.RedisCommandExecutionException: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Then I checked the redis server by logging in with redis-cli and found that ping was not working

(error) MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
The tips and their friendliness are long but I like them:)

Redis problem

MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Redis is configured to save a database snapshot, but it cannot be persisted to the hard disk at this time.The command used to modify the collection data is not available.Check the Redis log for detailed error information.

Reason

Forced closure of Redis snapshots resulted in non-persistence.It is essentially caused by insufficient memory.Therefore, related memory processing is required:

  1. Modify redis config
  2. Modify the directory of the snapshot backup to redirect the snapshot to another directory
  3. Modify host memory configuration

Solution 1

Set stop-writes-on-bgsave-error to no by closing save persistent snapshots directly

127.0.0.1:6379> config set stop-writes-on-bgsave-error no

There may be some problems with this in production, but this method won't solve the problem. He just lets us "ignore" him. Before using it, we need to confirm why bgsave failed. This is allowed when redis are used for caching, session scenarios, etc.

Solution 2

Redirect backed up rdb files to directory

CONFIG SET dir /tmp/some/directory/other/than/var
CONFIG SET dbfilename temp.rdb

After using this command, you need to ensure that bgsave_in_programs returns a result of 0

Solution 3

Dynamically modify the running parameters of the kernel while it is running

echo 'vm.overcommit_memory = 1' >> /etc/sysctl.conf
sysctl vm.overcommit_memory=1

Tags: Linux Redis Java snapshot Spring

Posted on Mon, 13 Jan 2020 15:16:22 -0500 by ubersmuck