riot

riot copied to clipboard

Hi, I tried to create a simple setup on my laptop. 2 Redis servers, one on port 6379 and another on port 6380. Then I set up live replication from on server to the other using the riot docker container. After that I inserted 500000 keys using the command cat set.txt | redis-cli --pipe. I'm not sure why riot processed only about 16000 keys.

Scanning/Listening 100% [=====================================================================================================] 16551/16551 (0:00:12 / 0:00:00) 1379.2/s

Source DB

127.0.0.1:6379> dbsize
(integer) 500000
127.0.0.1:6379> get test_20034
"value_20034"

Destination DB

127.0.0.1:6380> dbsize
(integer) 16551
127.0.0.1:6380> get test_20034
(nil)

Here are the complete logs from the riot container

 ~  docker run -it --rm --name riot --net host riotx/riot replicate --mode live redis://localhost:6379 redis://localhost:6380 --info    554ms  Wed Dec  4 13:23:11 2024
[main] INFO com.redis.riot.Replicate - Creating target Redis context with redis://localhost:6380 TargetRedisArgs(username=null, timeout=PT1M, tls=false, insecure=false, clientName=null, cluster=false, protocolVersion=RESP2, poolSize=8, readFrom=UPSTREAM) SslArgs(keystore=null, truststore=null, keyCert=null, key=null, trustedCerts=null)
[main] INFO com.redis.riot.Replicate - Creating source Redis context with redis://localhost SourceRedisArgs(username=null, timeout=PT1M, tls=false, insecure=false, clientName=null, cluster=false, protocolVersion=RESP2, poolSize=8, readFrom=UPSTREAM) SslArgs(keystore=null, truststore=null, keyCert=null, key=null, trustedCerts=null)
[main] INFO com.redis.riot.Replicate - Creating Redis dump reader
[main] INFO com.redis.riot.Replicate - Configuring RedisItemReader with RedisReaderArgs(mode=LIVE, keyPattern=null, keyType=null, scanCount=1000, queueCapacity=10000, threads=1, chunkSize=50, memUsageLimit=null, memUsageSamples=5, flushInterval=50, idleTimeout=0, eventQueueCapacity=10000, retryLimit=0, skipLimit=0, keyFilterArgs=KeyFilterArgs(includes=null, excludes=null, slots=null), pollTimeout=100)
[main] INFO com.redis.riot.Replicate - Creating Redis dump writer
[main] INFO com.redis.riot.Replicate - Configuring target Redis writer with RedisWriterArgs(multiExec=false, waitReplicas=0, waitTimeout=PT1S, merge=false)
[main] INFO com.redis.riot.Replicate - Configuring step with scan size estimator
[main] INFO com.redis.riot.Replicate - Retrieved config notify-keyspace-events: AKE
[main] INFO com.redis.riot.Replicate - Configuring export step with live true, flushInterval PT0.05S, idleTimeout PT2562047788015H12M55.807S
[main] INFO com.redis.riot.Replicate - Creating SpEL evaluation context with EvaluationContextArgs(varExpressions={}, dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSSZ, numberFormat=#,###.##, vars={})
[main] INFO com.redis.riot.Replicate - Creating processor with KeyValueProcessorArgs(keyExpression=null, typeExpression=null, ttlExpression=null, propagateTtl=true, propagateIds=true, prune=false)
[main] INFO com.redis.riot.Replicate - Creating Redis quick compare reader
[main] INFO com.redis.riot.Replicate - Configuring RedisItemReader with RedisReaderArgs(mode=LIVE, keyPattern=null, keyType=null, scanCount=1000, queueCapacity=10000, threads=1, chunkSize=50, memUsageLimit=null, memUsageSamples=5, flushInterval=50, idleTimeout=0, eventQueueCapacity=10000, retryLimit=0, skipLimit=0, keyFilterArgs=KeyFilterArgs(includes=null, excludes=null, slots=null), pollTimeout=100)
[main] INFO com.redis.riot.Replicate - Creating Redis quick compare reader
[main] INFO com.redis.riot.Replicate - Creating KeyComparator with ttlTolerance=PT0.1S ignoreStreamMessageId=false
[main] INFO com.redis.riot.Replicate - Creating SpEL evaluation context with EvaluationContextArgs(varExpressions={}, dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSSZ, numberFormat=#,###.##, vars={})
[main] INFO com.redis.riot.Replicate - Creating processor with KeyValueProcessorArgs(keyExpression=null, typeExpression=null, ttlExpression=null, propagateTtl=true, propagateIds=true, prune=false)
[main] INFO com.redis.riot.Replicate - Creating SpEL evaluation context with EvaluationContextArgs(varExpressions={}, dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSSZ, numberFormat=#,###.##, vars={})
[main] INFO com.redis.riot.Replicate - Creating processor with KeyValueProcessorArgs(keyExpression=null, typeExpression=null, ttlExpression=null, propagateTtl=true, propagateIds=true, prune=false)
[main] INFO com.redis.riot.Replicate - Creating Step(name=step, reader=com.redis.spring.batch.item.redis.RedisItemReader@311ff287, writer=com.redis.spring.batch.item.redis.RedisItemWriter@7377781e, taskName=Scanning/Listening, statusMessageSupplier=com.redis.riot.core.Step$$Lambda/0x0000002001326758@31db34da, maxItemCountSupplier=com.redis.spring.batch.item.redis.reader.RedisScanSizeEstimator@109f8c7e, processor=com.redis.riot.KeyValueFilter@5edacf20, executionListeners=[], readListeners=[], writeListeners=[], live=true, flushInterval=PT0.05S, idleTimeout=PT2562047788015H12M55.807S, skip=[], noSkip=[], retry=[], noRetry=[])
[main] INFO com.redis.riot.Replicate - Creating step replicate-step with chunk size 50
[main] INFO com.redis.riot.Replicate - Creating flushing step with flush interval PT0.05S and idle timeout PT2562047788015H12M55.807S
[main] INFO com.redis.riot.Replicate - Skipping fault-tolerance for step step
[main] INFO com.redis.riot.Replicate - Creating Step(name=compare, reader=com.redis.spring.batch.item.redis.reader.KeyComparisonItemReader@d3f4505, writer=com.redis.riot.CompareStatusItemWriter@3eb292cd, taskName=Comparing, statusMessageSupplier=com.redis.riot.AbstractReplicateCommand$$Lambda/0x0000002001341560@7fd987ef, maxItemCountSupplier=com.redis.spring.batch.item.redis.reader.RedisScanSizeEstimator@7209ffb5, processor=null, executionListeners=[com.redis.riot.CompareStepListener@7e9f2c32], readListeners=[], writeListeners=[], live=false, flushInterval=PT0.05S, idleTimeout=PT2562047788015H12M55.807S, skip=[], noSkip=[], retry=[], noRetry=[])
[main] INFO com.redis.riot.Replicate - Creating step replicate-compare with chunk size 50
[main] INFO com.redis.riot.Replicate - Skipping fault-tolerance for step compare
[main] INFO org.springframework.batch.core.launch.support.SimpleJobLauncher - Job: [SimpleJob: [name=replicate]] launched with the following parameters: [{}]
[main] INFO org.springframework.batch.core.job.SimpleStepHandler - Executing step: [replicate-step]
[main] INFO com.redis.spring.batch.item.redis.RedisItemReader - Creating queue with capacity 10,000
Scanning/Listening  ? % [] 0/0 (0:00:00 / ?) ?/s
[SimpleAsyncTaskExecutor-1] INFO org.springframework.batch.core.launch.support.SimpleJobLauncher - Job: [SimpleJob: [name=replicate-step-reader]] launched with the following parameters: [{}]
[SimpleAsyncTaskExecutor-1] INFO org.springframework.batch.core.job.SimpleStepHandler - Executing step: [replicate-step-reader]
[SimpleAsyncTaskExecutor-1] INFO com.redis.spring.batch.item.redis.reader.KeyScanEventItemReader - Creating queue with capacity 10,000
[SimpleAsyncTaskExecutor-1] INFO com.redis.spring.batch.item.redis.reader.RedisPubSubHandler - Establishing pub/sub connection to Redis
[SimpleAsyncTaskExecutor-1] INFO com.redis.spring.batch.item.redis.reader.RedisPubSubHandler - Subscribing to pattern __keyevent@0__:*
Scanning/Listening 100% [=====================================================================================================] 16551/16551 (0:00:12 / 0:00:00) 1379.2/s



I even tried setting `--threads 8 --read-threads 2` (my laptop has 8 cores)
Still it could replicate only 22k keys with increased threads

Scanning/Listening 100% [=====================================================================================================] 22350/22350 (0:00:10 / 0:00:00) 2235.0/s

Any idea what could be going on here? It's not replicating all the keys and also not reporting any errors. 🤷🏽‍♂️