bigkeys analyze support

[Phoeniwx]

trafficstars

Search before asking

• [X] I had searched in the issues and found no similar issues.

Motivation

We all know redis support --bigkeys & --memkeys commands, which can help us analyze bigkeys conveniently. Or we can use rdb generated by redis to parse bigkeys.

$ redis-cli --bigkeys

# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.01 to sleep 0.01 sec
# per SCAN command (not usually needed).

[00.00%] Biggest string found so far 'key-419' with 3 bytes
[05.14%] Biggest list   found so far 'mylist' with 100004 items
[35.77%] Biggest string found so far 'counter:__rand_int__' with 6 bytes
[73.91%] Biggest hash   found so far 'myobject' with 3 fields

However, according to my knowledge, kvrocks does not support --bigkeys or --memkeys, because of its different iteration way and other reasons, wshich makes it hard to analyze bigkeys.

Solution

Maybe you guys can add a --bigkeys(--memkeys) feature like redis or generate some kind of rdb-like file for us to analyze? thanks ; ).

Are you willing to submit a PR?

• [ ] I'm willing to submit a PR!

[mapleFU]

Seems that redis implement --bigkey as a client tool: https://github.com/redis/redis/blob/unstable/src/redis-cli.c#L8257 , it will send scan to sample the bigkeys.

Maybe I can try to test using --bigkey in client in next few weeks

[git-hulk]

Thanks to @mapleFU, it's true that Redis implements its big key detection in the redis-cli with the scan command. The reason why Kvrocks don't support the redis-cli --bigkey is we now return the string as scan iterator cursor instead of an integer, so it can't be compatible with redis-cli now

[jihuayu]

After I fix https://github.com/apache/incubator-kvrocks/issues/1402, redis-cli --bigkeys is work. But redis-cli uses DBSIZE to get the total number of keys and calculates the progress based on this information. So before we run DBSIZE scan, dbsize is 0, progress bar wiil display [ -nan%] or [ inf%]. We need to fix this issue? @git-hulk

[git-hulk]

@jihuayu

Cool, I think it's fine to display [-nan%] or [inf%], and ask users to execute the dbsize scan command if they wanna see the percent.

[jihuayu]

@git-hulk These texts are completely from redis-cli and we cannot control them. I think we can add the FAQ to our doc.

[git-hulk]

@git-hulk These texts are completely from redis-cli and we cannot control them. I think we can add the FAQ to our doc.

Yes, we can guide them via the documentation.