[NEW] Feature to find bigkeys

[otheng03]

Hi, valkey friends.

The problem/use-case that the feature addresses

When querying data from a hash with a very large number of elements using HGETALL, Valkey becomes slow while executing the command, which decreases the quality of service. Support for bigkeylog will enable user to check out which keys need to be optimized. Not like the valkey-cli --bigkeys, these commands will enable us to check it out without scanning the whole node.

Description of the feature

Add BIGKEY subcommand to commandlog

Part 1: Two new parameters in valkey.conf

# Logs keys that exceed a specified number of elements, or the length of the string if the key is a string
commandlog-bigkey-larger-than

# The length for bigkeylog entries.
commandlog-large-bigkey-max-len

There are 2 requirements for these 2 parameters:

1. These 2 parameters are allowed to be updated during runtime
2. For each BIGKEY LOG entry, at least 4 fields are included: entry_ID, Key name, the number of items for the key or string length, timestamp

Part 2: 3 new commands

127.0.0.1:6379> COMMANDLOG GET -1 BIGKEY
1) 1) (integer) 0
   2) (integer) 1743940262           
   3) (integer) 5
   4) 1) "hgetall"                     
      2) "h"
   5) "127.0.0.1:45368"
   6) ""

127.0.0.1:6379> COMMANDLOG LEN BIGKEY
(integer) 1

127.0.0.1:6379> COMMANDLOG RESET BIGKEY
OK

Demo implementation of bigkey command: #1151

Alternatives you've considered

Anyway, it would be nice if valkey could easily detect whether a bigkey exists or not. https://github.com/redis/redis/pull/13592 can be an alternative!

Considerations

It might be worth considering an option to configure thresholds separately for each data type.

Thanks for reading.

[otheng03]

Any comments on this would be greatly appreciated.

[wuranxx]

I think this is a valuable feature. Hot keys also have this requirement.

In now, the hot key function requires the server to use the lfu evicting policy and scan all keys to find the key with the highest frequency. This has a significant impact on valkey's performance.

[wuranxx]

I took a brief look the PR, has some question:

1. bigkey is not like slowlog, key size will be change when set command or some change element command, how to detect these behavior?
2. This PR like seems to add a bigkey history, which may have been a big key at some point in time, but now it may not be a big key.

[otheng03]

@wuranxx Thanks for your comment.

bigkey is not like slowlog, key size will be change when set command or some change element command, how to detect these behavior?

Keeping the Top N keys might be more useful. Handling the most impactful keys in the system could help identify and manage big keys more effectively.

However, maintaining Top N keys requires some computational complexity. So, the current version is designed to provide recently accessed big keys, similar to how slowlog works.

This PR like seems to add a bigkey history, which may have been a big key at some point in time, but now it may not be a big key.

That's right. I think if we update the big key log when a key size decreases, we could provide more accurate information.

In current version, since the smaller keys will no longer be maintained, the bigkey log will naturally be filled with the more recently used bigkeys.

[hwware]

In issue https://github.com/valkey-io/valkey/issues/1880, I already mention your idea. I think your PR is good start for pulling some memory information.

[otheng03]

@hwware Thank you for reviewing the PR and referencing the issue! Would you prefer implementing Feature Proposal 1 and 2 directly, or would it be better for me to continue developing the PR from #1151 with your feedback? I’m happy to revise my PR if needed. Either approach works for me, but since contributing to Valkey is a great experience, I’d love to continue working on #1151 if that’s okay with you.

[hwware]

My idea solution is like redis INFO keysizes solution as below:

db0_distrib_strings_sizes:1=19,2=655,4=3918,8=19,16=23,32=326,64=5,128=1,256=47,512=100899,1K=31,2K=29,4K=23,8K=16,16K=3,32K=2 db0_distrib_lists_items:1=5784492,2=151535,4=46670,8=20453,16=8637,32=3558,64=1047,128=676,256=533,512=218,4K=1,8K=42 db0_distrib_sets_items:1=7355615,2=207181,4=50612,8=21462,16=8864,32=2905,64=1365,128=974,256=773,512=675,1K=395,2K=292,4K=154,8K=89,16K=48,32K=21,64K=27,128K=16,256K=5,512K=4,1M=2 db0_distrib_hashes_items:2=1,4=544,8=746485,16=324174,32=141169,64=207329,128=4349,256=136226,1K=1

For string datatype, record how many memories the value is used, and for the other datatype, such as list, set, hash and sorted set, record how many elements the key has. The result is very close to above result. And then because user can fetch this information through INFO command, and we could let users or clients to decide which threshold (such as 64, or 1K, or 128k) is bigkeys.

How do you think?

[otheng03]

Managing key sizes in a distributed fashion is a good idea, as it gives us a clear picture of the overall key distribution.

[hwware]

Do you want to contribute this feature?

[otheng03]

Your idea aligns closely with the alternatives I’ve been considering in the 'Alternatives Considered' section, and I completely agree with your suggestion. Yes, I'm interested in contributing and would like to know what kind of contribution you think would be most helpful.

[hwware]

Thanks for your interest. I will write a more detail requirements for this bigkeys feature, maybe it takes 2-3 days. And I will update here, and then we can discuss and cooperate to work on this. Thanks

[hwware]

I have the following requirements for the BIGKEYS LOG, please take a look and update your top description in this issue and your PR, Thanks,

Part 1:

As your suggestion, you can define two new parameters in valkey.conf Specifies the minimum number of elements a bigkey must have before it gets logged into the bigkey log. #bigkeylog-num-elements-larger-than Defines the size of the bucket for storing bigkey log entries. #bigkeylog-bucket-size

In Valkey 8.1, there are new COMMAND LOG part in valkey.conf, which includes commandlog-request-larger-than, commandlog-execution-slower-than etc, or Maybe you can reference them to define the parameter name for BIGKEY

There are 2 requirements for these 2 parameters: #1. These 2 parameters are allowed to be updated during runtime #2. I suggest for each BIGKEY LOG entry, at least 4 fields are included: entry_ID, Key name, the number of items for the key or string length, timestamp

Part 2:

As what you mentioend, 3 new commands could be added: 127.0.0.1:6379> bigkeylog get

1. 1. "hash_test" # key
   2. (integer) 10689 # number of elements of the bigkey
   3. (integer) 1728659704 # Unix time at which the bigkey was grown.

127.0.0.1:6379> bigkeylog len (integer) 1

127.0.0.1:6379> bigkeylog reset OK

Please let me know if you have any concern.
Before you resume your working, I suggest you first rebase your PR first. Thanks,

BTW: I will work on INFO KEYSIZES part.

[otheng03]

Thank you for providing the detailed requirements. I understand them clearly. I’ll update the description at the top of this issue and rebase my PR(#1151) accordingly to align with the requirements.

[otheng03]

I'm considering between two methods. I would appreciate your comments on Option 1 and Option 2.

Option 1: Add BIGKEY as a subcommand of COMMANDLOG BIGKEYLOG records access logs when a user command accesses a big key. Since this logging mechanism is similar to how COMMANDLOG works, I thought it might make sense to implement it as a subcommand under COMMANDLOG. I updated the description at the top of this issue and revised and rebased #1151. I believe submitting a PR could help explain my idea in more detail.

Pros: Straight forward to understand BIGKEY command since it's almost the same with commandlog

Option 2: Create a separate BIGKEYLOG and store bigkey in hash bucket https://github.com/otheng03/valkey/blob/bigkeylog-backup/src/bigkeylog.c#L13-L30)

Pros: this approach can leave as much keys as possible by storing the keys in a bucket

Thanks!

[otheng03]

Let me try Option 2, if you are okay. :)

[ranshid]

@otheng03 As I agree the feature proposal to have a server-side tool to track down big/large keys is important, I am not sure I understand the details of this proposal. looking at https://github.com/valkey-io/valkey/pull/1151 it seems you mainly provided a way to track down hashes with many elements (which does not directly indicate the memory consumption of these objects right?) Going back to the problem statement:

When querying data from a hash with a very large number of elements using HGETALL, Valkey becomes slow while executing the command, which decreases the quality of service.

This indeed suggest a performance issue when the number of elements is very large. However users often would like to track large objects which are consuming lots of memory so they can delete them or debug/understand why their data is not balanced correctly. However calculating the memory size of an object might be expensive for these objects so I guess a memory tracking mechanism would have to be added for these object types (I recall we did that for rax)

Support for bigkeylog will enable user to check out which keys need to be optimized. Not like the valkey-cli --bigkeys, these commands will enable us to check it out without scanning the whole node.

Can you please explain what "optimized" means in this context? does it mean user would choose to use hscan instead? he can do that anyway and limit the output with the count right?

[otheng03]

Hi, @ranshid

Thanks for reading this topic.

Can you please explain what "optimized" means in this context? does it mean user would choose to use hscan instead? he can do that anyway and limit the output with the count right?

#1151 has been rebased once since I first created it, so it's now using commandlog instead. Sorry for the confusion. You can check the initial version before the rebase here https://github.com/otheng03/valkey/commit/e19f7eee8fdce8cd9cf219608f2a8d85ceadaef3

To briefly explain the initial version — it creats hash buckets with a fixed size, and if the number of elements is greater than bigkeylog-num-elements-larger-than, it stored the key and the number of elements in the hash bucket(LINK). (This is a draft I wrote to explain my idea, and currently it only supports the hash type and don't track deletion. It’s definitely possible to support other data types as well, but I was planning to handle that after we finalize the discussion and decide to implement the feature.)

(I recall we did that for rax)

I think calculating the total bytes is a good approach. I will read the link rax https://github.com/valkey-io/valkey/pull/688 when I get home from work.

[ranshid]

Thank you @otheng03. I am still not sure I understand how the user will react or use this data:

Support for bigkeylog will enable user to check out which keys need to be optimized.

Can you please provide some example use case of a client using this data to perform an "optimization" based on the size of the key?

[otheng03]

Thanks, I will leave comment after I get home. I might arrive home after 5 hours. Sorry for the delay in my response.

[otheng03]

@ranshid

Let me start by explaining the idea of https://github.com/otheng03/valkey/commit/e19f7eee8fdce8cd9cf219608f2a8d85ceadaef3. You can skip straight to the last part.

The method for storing a big key and the number of elements

When adding or removing elements from a hash, list, set, or zset, and etc the key and the number of elements are stored in a bucket if the number of elements are larger than bigkeylog-num-elements-larger-than. (in the PR their is lack of features, but this comment just explain my idea) If a hash collision occurs, the existing entry is simply overwrriten. The number of buckets is fixed based on a configured value. (Assumed that hash(key1) = 1, hash(key2) = 2, hash(key3) = 10)

• Pros
  • time complexity is O(1)
    • since getting length of a key O(1), hash(key) is O(1), and putting (key, num_elements) into the bucket is O(1)
• Con
  • An item in the bucket is overwritten when two or more key hashes collide. However, if the user sets a larger bucket size by increasing the max-len value, or resolves the big keys, they will eventually be able to find the big key causing problem.

Usage Example command :

$ valkey-cli

127.0.0.1:6379> BIGKEYLOG GET 
1) 1) (integer) 0   // Entry ID
   2) (integer) key1  // key name        
   3) (integer) 10000 // the number of elements
   4) (integer) 1743940262 // last updated timestamp
2) 1) (integer) 1   
   2) (integer) key2 
   3) (integer) 5000 
   4) (integer) 1743940462 
3) 1) (integer) 2   
   2) (integer) key3
   3) (integer) 25000 
   4) (integer) 1743940462 

Can you please provide some example use case of a client using this data to perform an "optimization" based on the size of the key?

BIGKEYLOG GET provides users with a list of recently accessed big keys that exist in the system. Keys that exceed the configured threshold can negatively affect system performance when operations require accessing almost of their elements. As the workload of a user's service changes, unexpectedly large keys can become a risk factor that impacts performance. (Users may not know how large their stored hashes or lists can grow — especially in cases caused by abusers or system bugs.) By using BIGKEYLOG, users can identify and optimize keys that store more elements than expected in advance, helping to prevent potential issues.

If there is any other ambiguity, please let me know. Thanks.

[hwware]

@otheng03 As I agree the feature proposal to have a server-side tool to track down big/large keys is important, I am not sure I understand the details of this proposal. looking at #1151 it seems you mainly provided a way to track down hashes with many elements (which does not directly indicate the memory consumption of these objects right?) Going back to the problem statement:

When querying data from a hash with a very large number of elements using HGETALL, Valkey becomes slow while executing the command, which decreases the quality of service.

This indeed suggest a performance issue when the number of elements is very large. However users often would like to track large objects which are consuming lots of memory so they can delete them or debug/understand why their data is not balanced correctly. However calculating the memory size of an object might be expensive for these objects so I guess a memory tracking mechanism would have to be added for these object types (I recall we did that for rax)

Support for bigkeylog will enable user to check out which keys need to be optimized. Not like the valkey-cli --bigkeys, these commands will enable us to check it out without scanning the whole node.

Can you please explain what "optimized" means in this context? does it mean user would choose to use hscan instead? he can do that anyway and limit the output with the count right?

Hi Ran, in fact, there are 3 different terms in valkey-cli optional parameters:

1. big keys: It calculates how many items the key has or the string length of the value for the string datatype. It is the result from https://github.com/redis/redis/pull/13592 (I am working on this)
2. large key: This is like what you mention, it is to calculate how many memory consumption for the keys. It is very cost, thus we have no good way to implement this feature so far. (But It is on the roadmap)
3. hot key: This is to calculate the access frequency for a key in a specific time.

I think this issue only relates to big keys, because some of my clients are interested in the big key information from time to time, such as they pull the INFO command every 10 seconds.

[ranshid]

As the workload of a user's service changes, unexpectedly large keys can become a risk factor that impacts performance. (Users may not know how large their stored hashes or lists can grow — especially in cases caused by abusers or system bugs.) By using BIGKEYLOG, users can identify and optimize keys that store more elements than expected in advance, helping to prevent potential issues.

So if I understand correctly this will be used mainly for debugging and apply mitigations when user experience issues (or in order to setup automatic repair mechanisms) which I think is fine. I do think we need to clear about the "big" semantic.

if we treat "big" as the number of contained elements (more relevant for complex datatypes) it can help solve some cases you mentioned. I do not think the HGETALL example is very good, simply because it does not follow the best practices which prefer using HSCAN. I do think performance issues can rise from other examples like ZSET which has logarithmic dependency on the number of elements, thus it might be helpful to locate it. I think usually the object total used memory can provide a good estimation to the complexity and would also fit better for module defined objects. I understand you are following this route now so will wait to catch the changes.

Thank you for following up on that @otheng03!

[ranshid]

@otheng03 As I agree the feature proposal to have a server-side tool to track down big/large keys is important, I am not sure I understand the details of this proposal. looking at #1151 it seems you mainly provided a way to track down hashes with many elements (which does not directly indicate the memory consumption of these objects right?) Going back to the problem statement:

When querying data from a hash with a very large number of elements using HGETALL, Valkey becomes slow while executing the command, which decreases the quality of service.

This indeed suggest a performance issue when the number of elements is very large. However users often would like to track large objects which are consuming lots of memory so they can delete them or debug/understand why their data is not balanced correctly. However calculating the memory size of an object might be expensive for these objects so I guess a memory tracking mechanism would have to be added for these object types (I recall we did that for rax)

Support for bigkeylog will enable user to check out which keys need to be optimized. Not like the valkey-cli --bigkeys, these commands will enable us to check it out without scanning the whole node.

Can you please explain what "optimized" means in this context? does it mean user would choose to use hscan instead? he can do that anyway and limit the output with the count right?

Hi Ran, in fact, there are 3 different terms in valkey-cli optional parameters:

1. big keys: It calculates how many items the key has or the string length of the value for the string datatype. It is the result from Add KEYSIZES section to INFO redis/redis#13592 (I am working on this)
2. large key: This is like what you mention, it is to calculate how many memory consumption for the keys. It is very cost, thus we have no good way to implement this feature so far. (But It is on the roadmap)
3. hot key: This is to calculate the access frequency for a key in a specific time.

I think this issue only relates to big keys, because some of my clients are interested in the big key information from time to time, such as they pull the INFO command every 10 seconds.

I am still not sure why the "big keys" metric is very important. First it might require Modules to support providing this info for their new data types (eg JSON to provide the number of JSON items?) Seconds I think Object memory can provide a good estimation for the complexity. I agree there are some cases (eg ZSET) which might be impacted by the number of elements even when the elements are small and the overall memory utilization is low, but I think this is an acceptable compromise.

Regarding memory utilization calculations: for complex elements (no regular strings) we can calculate it by following allocations (like we did for the RAX type). It might not be trivial to do, but it should provide a very fast query ability

[hwware]

I could provide you some background information:

Now when server calculates used memory and decide if key eviction process should begin, we do not calculate the total memory consumption for all the keys in memory, we instead use a reverse calculation -- used_memory - overhead_memory, because it is too expensive to calculate the memory usage of every key directly. So it is same, implementing large keys in real time is challenging, thus someone use the big keys as an alternative as an approximate method to achieve the large key goal.

If you have a good way to calculate the key usage in real time, welcome to share with us. Thanks

[ranshid]

I agree that maintaining some list/hash of largest keys is a good idea, no argue about that. I only meant that we should base this only on key memory consumption and avoid mixing a new extra notion of "big key" which AFAIU is meant to be the key "cardinality" and not "size". IMO by maintaining a memory size tracking per keys is a way to be able to fast calculate the size after each key "mutation" and be able to modify a tracking list (which is what the PR did only for "cardinality" and not size)

[otheng03]

Thank you both. IIUC, we all agree on adding a feature to maintain a list of logs or hash bucket of the large keys. I’ll start by trying the hash bucket approach first.

Regarding the meaning of "big", the valkey-cli documentation says that bigkeys refers to keys with many elements, while memkeys refers to keys that consume a lot of memory.

Reference: https://valkey.io/topics/cli/

--bigkeys : Sample keys looking for keys with many elements (complexity).
--memkeys : Sample keys looking for keys consuming a lot of memory.

The current meaning is somewhat ambiguous.

To be honest, I'd like to support both conditions: the number of elements and memory usage. Two conditions are useful. Since checking the number of elements is straightforward, it’s a simple way to spot problems in the data — such as when there’s a limit on the number of friends that can be registered. In some cases, the actual memory usage may be more important than the number of elements.

Therefore, I'd like to implement the first version of this feature as described below. I don’t have a strong opinion on it, so please feel free to share any ideas or suggestions.

Two new parameters in valkey.conf

# Log keys to the bucket when the number of elements exceeds a specified threshold, or when the key is a string and its length exceeds the threshold.
keyinfo-num-elements-larger-than

# The length for bigkeylog entries.
keyinfo-large-num-elements-max-len

Three commands

KEYINFO MANY_ELEMENTS <count> GET
KEYINFO MANY_ELEMENTS LEN
KEYINFO MANY_ELEMENTS RESET

After implementing the first version, we can consider adding one more feature.

# Log keys to the bucket when they exceed a specified threshold in memory usage or element count.
keyinfo-memory-usage-larger-than

# The length for entries.
keyinfo-large-memory-usage-max-len

KEYINFO LARGE_MEMORY <count> GET
KEYINFO LARGE_MEMORY LEN
KEYINFO LARGE_MEMORY RESET

[hwware]

So you mean you want to implement 2 features: both bigkeys and large keys? From my suggestion, let us first work on bigkeys feature, because large keys could be tricky. How do you think?

[otheng03]

Thanks, I totally agree. I'll start implementing the first feature below:

- Add two new parameters to valkey.conf:
  - keyinfo-num-elements-larger-than
  - keyinfo-large-num-elements-max-len
- Implement three new commands:
  - KEYINFO MANY_ELEMENTS <count> GET
  - KEYINFO MANY_ELEMENTS LEN
  - KEYINFO MANY_ELEMENTS RESET

Once the first feature is complete, I’d be happy if we could also consider supporting the large keys(=keys that use a large amount of memory) feature. As ranshid kindly mentioned, large keys would be a valuable feature to have.

[otheng03]

#1151 is ready for review. Since this is my first time contributing to a feature, I’m sure there’s room for improvement. I’d appreciate it if you could take a look when you have time.