READONLY You can't write against a read only replica

[geekthread]

Hi Team,

We have been recently getting redis read only replica errors frequently whenever we are doing upgrades on of our redis dependent services.

We have also checked redis andsentinel logs for a probable failover/errors but there was nothing in the logs and redis setup looks healthy (1 Master and 2 Slaves , All Sentinels up)

We have below queries:

1. Is there any way to find the root cause of this issue ?
2. Is there any way to debug this issue ?
3. Is it possible that issue is related to network outage ? (We are on GCP)

Please find the stack trace :
{ "org.springframework.data.redis.connection.RedisPipelineException: Pipeline contained one or more invalid commands; nested exception is org.springframework.dao.InvalidDataAccessApiUsageException: READONLY You can't write against a read only replica.; nested exception is redis.clients.jedis.exceptions.JedisDataException: READONLY You can't write against a read only replica." : [["JedisConnection.java",398,"org.springframework.data.redis.connection.jedis.JedisConnection.convertPipelineResults"],["JedisConnection.java",363,"org.springframework.data.redis.connection.jedis.JedisConnection.closePipeline"],["",-1,"sun.reflect.GeneratedMethodAccessor815.invoke"],["DelegatingMethodAccessorImpl.java",43,"sun.reflect.DelegatingMethodAccessorImpl.invoke"],["Method.java",498,"java.lang.reflect.Method.invoke"],["CloseSuppressingInvocationHandler.java",61,"org.springframework.data.redis.core.CloseSuppressingInvocationHandler.invoke"],["",-1,"com.sun.proxy.$Proxy1187.closePipeline"],["XyzRedisTemplate.java",62,"com.xyz.cache.impl.configuration.XyzRedisTemplate.lambda$executePipelined$1"],["RedisTemplate.java",228,"org.springframework.data.redis.core.RedisTemplate.execute"],["RedisTemplate.java",188,"org.springframework.data.redis.core.RedisTemplate.execute"],["RedisTemplate.java",175,"org.springframework.data.redis.core.RedisTemplate.execute"],["XyzRedisTemplate.java",53,"com.xyz.cache.impl.configuration.XyzRedisTemplate.executePipelined"],["RedisTemplate.java",314,"org.springframework.data.redis.core.RedisTemplate.executePipelined"],["RedisOperations.java",228,"com.xyz.cache.impl.store.distributed.redis.RedisOperations.lambda$nonReturningOperations$12"],["RedisOperations.java",236,"com.xyz.cache.impl.store.distributed.redis.RedisOperations.performOperation"],["RedisOperations.java",230,"com.xyz.cache.impl.store.distributed.redis.RedisOperations.nonReturningOperations"],["DistributedCache.java",508,"com.xyz.cache.impl.store.distributed.redis.DistributedCache.evictInternalMultipleItems"],["DistributedCache.java",503,"com.xyz.cache.impl.store.distributed.redis.DistributedCache.evictInternalMultiple"],["DistributedCache.java",450,"com.xyz.cache.impl.store.distributed.redis.DistributedCache.evictInternal"],["DistributedStaleCacheEvictionService.java",147,"com.xyz.cache.impl.store.DistributedStaleCacheEvictionService.evictStaleKeys"],["DistributedStaleCacheEvictionService.java",82,"com.xyz.cache.impl.store.DistributedStaleCacheEvictionService.lambda$submit$0"],["Executors.java",511,"java.util.concurrent.Executors$RunnableAdapter.call"],["FutureTask.java",266,"java.util.concurrent.FutureTask.run"],["ThreadPoolExecutor.java",1149,"java.util.concurrent.ThreadPoolExecutor.runWorker"],["ThreadPoolExecutor.java",624,"java.util.concurrent.ThreadPoolExecutor$Worker.run"],["Thread.java",748,"java.lang.Thread.run"]],"Caused By": { "org.springframework.dao.InvalidDataAccessApiUsageException: READONLY You can't write against a read only replica.; nested exception is redis.clients.jedis.exceptions.JedisDataException: READONLY You can't write against a read only replica." : ["JedisExceptionConverter.java",69,"org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert","JedisExceptionConverter.java",42,"org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert","PassThroughExceptionTranslationStrategy.java",44,"org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate","FallbackExceptionTranslationStrategy.java",42,"org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate","JedisConnection.java",135,"org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException","JedisConnection.java",385,"org.springframework.data.redis.connection.jedis.JedisConnection.convertPipelineResults"],"Caused By": { "redis.clients.jedis.exceptions.JedisDataException: READONLY You can't write against a read only replica." : ["Protocol.java",132,"redis.clients.jedis.Protocol.processError","Protocol.java",166,"redis.clients.jedis.Protocol.process","Protocol.java",220,"redis.clients.jedis.Protocol.read","Connection.java",318,"redis.clients.jedis.Connection.readProtocolWithCheckingBroken","Connection.java",330,"redis.clients.jedis.Connection.getMany","Pipeline.java",98,"redis.clients.jedis.Pipeline.sync","JedisConnection.java",374,"org.springframework.data.redis.connection.jedis.JedisConnection.convertPipelineResults"]}}} }

[Queetinliu]

execute info replication on redis and check the output,does the master_link_status up?log to master redis and execute role does it shows master?check redis log does the redis rsync look right?