riot
riot copied to clipboard
redis
redis bigkey issue
hello thanks for great great tool!
I have some issues using this tool, so I'm asking this question.
The following error occurs when there is a big key in redis. What action can be taken in this case?
my command
riot-redis --info -h dbatest-source -p 6379 replicate --type dump -h dbatest-target -p 6379 --mode live --reader-queue=50000
Bigkey information is as follows.
-------- summary -------
Sampled 3 keys in the keyspace!
Total key length in bytes is 17 (avg len 5.67)
Biggest hash found '"myhash"' has 3392811 fields
Biggest set found '"myset"' has 3392251 members
Biggest zset found '"myzset"' has 3392127 members
0 lists with 0 items (00.00% of keys, avg size 0.00)
1 hashs with 3392811 fields (33.33% of keys, avg size 3392811.00)
0 strings with 0 bytes (00.00% of keys, avg size 0.00)
0 streams with 0 entries (00.00% of keys, avg size 0.00)
1 sets with 3392251 members (33.33% of keys, avg size 3392251.00)
1 zsets with 3392127 members (33.33% of keys, avg size 3392127.00)
riot-redis error
type #1 - migrate hash key
Executing step: [redis-scan-reader]
Scanning ? % ││ 0/0 (0:00:00 / ?) ?/s
Job redis-scan-reader status: STARTED
Encountered an error executing step redis-scan-reader in job redis-scan-reader: minimumReadableBytes : -824121627 (expected: >= 0)
Step: [redis-scan-reader] executed in 26s428ms
Closing connection pool
Job: [SimpleJob: [name=redis-scan-reader]] completed with the following parameters: [{}] and the following status: [FAILED] in 26s435ms
Scanning ? % ││ 0/0 (0:00:26 / ?) .0/s
Step: [scan-replication-step] executed in 32s293ms
Closing redis-scan-reader
Closing connection pool
type #2
Executing step: [redis-scan-reader]
Scanning 0% │ │ 0/4153281 (0:00:00 / ?) ?/s
Scanning 75% │████████████████████████████████████████████████████████ │ 3143600/4153281 (0:10:20 / 0:03:19) 5070.3/s
Unexpected exception during request: java.io.IOException: Protocol wrong type for socket
Unexpected exception during request: java.io.IOException: Protocol wrong type for socket
Unexpected exception during request: java.io.IOException: Protocol wrong type for socket
.
.
.
Unexpected exception during request: java.io.IOException: Protocol wrong type for socket
Reconnecting, last destination was dbatest-target:6379
The above error occurred when I met a big key during normal migration( other string keys were migrated successfully).
please help !
@jruaux Hi, I can't send you the rdb dump file because it's elasticache. But the dataset I created looks like this.
redis-benchmark -h dbatest-source -n 5000000 -r 6000000 -d 1000 -c 1000 -q -t set,incr,sadd,hset,zadd
In particular, a problem occurred in the data of hset. There was no difference between dump and ds of --type option.