SocketTimeoutException on deletion of a million keys with deleteAll

[florian2412]

redis-om-spring in 0.8.0 spring-boot in 3.0.4

Hi !

We have in our redis database about one million Json data with one Json = one key.

If we delete all the keys at once with the deleteAll of the RedisDocumentRepository, we have a SocketTimeoutException. After analysis, we see that the objects are all deleted but the command dropIndex carried out crashes and this has the consequence of lost the index.

In RedisInsight / Analysis tool / Slow log, we can see :

1. _FT.SEARCH our.index * WITHSCORES LIMIT 0 0 in 264 025 µs
2. FT.SEARCH our.index * LIMIT 0 0 in 15 µs
3. FT.DROPINDEX our.index DD in 9 326 607 µs

And the log :

redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out] with root causejava.net.SocketTimeoutException: Read timed out at java.base/sun.nio.ch.NioSocketImpl.timedRead(Unknown Source) ~[na:na] at java.base/sun.nio.ch.NioSocketImpl.implRead(Unknown Source) ~[na:na] at java.base/sun.nio.ch.NioSocketImpl.read(Unknown Source) ~[na:na] at java.base/sun.nio.ch.NioSocketImpl$1.read(Unknown Source) ~[na:na] at java.base/java.net.Socket$SocketInputStream.read(Unknown Source) ~[na:na] at java.base/java.io.InputStream.read(Unknown Source) ~[na:na] at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:202) ~[jedis-4.3.1.jar:na] at redis.clients.jedis.util.RedisInputStream.readByte(RedisInputStream.java:46) ~[jedis-4.3.1.jar:na] at redis.clients.jedis.Protocol.process(Protocol.java:126) ~[jedis-4.3.1.jar:na] at redis.clients.jedis.Protocol.read(Protocol.java:192) ~[jedis-4.3.1.jar:na] at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:316) ~[jedis-4.3.1.jar:na] at redis.clients.jedis.Connection.getOne(Connection.java:298) ~[jedis-4.3.1.jar:na] at redis.clients.jedis.Connection.executeCommand(Connection.java:123) ~[jedis-4.3.1.jar:na] at redis.clients.jedis.executors.DefaultCommandExecutor.executeCommand(DefaultCommandExecutor.java:24) ~[jedis-4.3.1.jar:na] at redis.clients.jedis.UnifiedJedis.executeCommand(UnifiedJedis.java:167) ~[jedis-4.3.1.jar:na] at redis.clients.jedis.UnifiedJedis.ftDropIndexDD(UnifiedJedis.java:3521) ~[jedis-4.3.1.jar:na] at com.redis.om.spring.ops.search.SearchOperationsImpl.dropIndexAndDocuments(SearchOperationsImpl.java:86) ~[redis-om-spring-0.8.0.jar:na] at com.redis.om.spring.RedisJSONKeyValueAdapter.deleteAllOf(RedisJSONKeyValueAdapter.java:210) ~[redis-om-spring-0.8.0.jar:na] at org.springframework.data.keyvalue.core.KeyValueTemplate.lambda$delete$4(KeyValueTemplate.java:266) ~[spring-data-keyvalue-3.0.3.jar:3.0.3] at org.springframework.data.keyvalue.core.KeyValueTemplate.execute(KeyValueTemplate.java:314) ~[spring-data-keyvalue-3.0.3.jar:3.0.3] at org.springframework.data.keyvalue.core.KeyValueTemplate.delete(KeyValueTemplate.java:264) ~[spring-data-keyvalue-3.0.3.jar:3.0.3] at org.springframework.data.keyvalue.repository.support.SimpleKeyValueRepository.deleteAll(SimpleKeyValueRepository.java:162) ~[spring-data-keyvalue-3.0.3.jar:3.0.3]

We found a workaround using a jedis pipeline to delete all the objects without deleting the index.

[bsbodden]

@sazzad16 @maorohana-redis What do you guys think? I went with dropping the index for deleteAll because simply deleting the entries causes reindexing, which is expensive and time-consuming, so queries against large collections are inaccurate until the indexing process is finished. That is weird that dropIndex is the failure here.

[maorohana-redis]

@bsbodden I see that we call to ftDropIndexDD which will delete all the associated documents so it’s O(N) in this case and can describe the timeout exception .. I think that we need to call ftDropIndex without DD flag and remove all the keys in pipeline mode , WDYT?

[bsbodden]

@bsbodden I see that we call to ftDropIndexDD which will delete all the associated documents so it’s O(N) in this case and can describe the timeout exception .. I think that we need to call ftDropIndex without DD flag and remove all the keys in pipeline mode , WDYT?

That sounds like a good test, we should profile it and see which one is more efficient/atomic!

[bsbodden]

I see the default Jedis timeout is 2000 in https://github.com/redis/jedis/blob/master/src/main/java/redis/clients/jedis/Protocol.java#L22 @florian2412 can you try the configuration spring.redis.timeout=15000

[bsbodden]

@sazzad16 can we send/set a socket timeout just for a specific command?

[sazzad16]

@bsbodden setting socket timeout is possible in Connection class by setSoTimeout(int soTimeout) method.

[sazzad16]

In Connection object, you can do setTimeoutInfinite() before operations (assuming blockingTimeoutMillis is large enough). You should rollbackTimeout() after operations in this case.

[sazzad16]

You can also use sendBlockingCommand in Jedis/UnifiedJedis.

[sazzad16]

Even more, you can use executeCommand with blocking() set in CommandArguments.