[Feature Request]:scan all keys

[joeleclems]

Problem description (Describe the problem you would like to solve with this feature request or suggested idea). i would like to scan ALL keys of database

How often do you encounter this problem (Describe how frequently this problem occurs). always : our databases have more than 10k keys : so i have to clic on "Scan more" button a lot of time

Alternatives considered (Describe any alternative features or products you've considered). clic on "Scan more" button a lot of time

Additional information (Add any other context or details here).

[ViktarStarastsenka]

Hello @joeleclems , would you like to scan all keys of your database when you filter per key type/key name or get all the keys in the Tree view?

[joeleclems]

Hello @ViktarStarastsenka , all the keys in the Tree view. Thanks

[ViktarStarastsenka]

@joeleclems , how many keys do you have in the database? there is a risk that you may get the "out of memory" error.

[joeleclems]

@ViktarStarastsenka it depend. From 10 000 to 1 000 000

[mvastola]

I have the same feature request. We have ~63M keys. Not sure how much memory that will take, but I don't have a problem with it tying up my computer for a while

[ViktarStarastsenka]

@mvastola , can you please describe your use case? I believe it is not likely that you want to view 63M keys.

[mvastola]

@ViktarStarastsenka we noticed a sudden uptick (around 20%) in the amount of keys currently in redis. We want to investigate which key prefix is responsible so we can identify where they're being set and make sure nothing is wrong.

[ViktarStarastsenka]

@mvastola , sorry for the delay, but I have a concern that it is possible to use the Tree view for this purpose, since all the keys are grouped by FE and it will reach OOM much sooner than analysing 63M keys. Did you try the "Memory Analysis" tool in RedisInsight V1?

[ViktarStarastsenka]

@joeleclems , can you please provide additional information about your use case?

[joeleclems]

@ViktarStarastsenka sorry for the delay, my use case is to have an tree folder like windows, Rdis Desktop Manager etc. A ggod example of what we would like :

An other exemple :

• a1:b1:c1
• a1:b2:c3
• a2:b2:c2 When i load "all keys", i need to only see :
• a1 (2) => the "2" means there is 2 keys starting with a1 in my database
• a2 When i clic a1, i need to see : -a1 --a1:b1 --a1:b2 -a2

So, to resume, i would like to see all the keys by "key level"

[mohan-venkatakrishnan]

Hi @ViktarStarastsenka

In V2 Memory analysis, we are not able to scan more than 10000 Keys. I have a dump of 6M keys and want to perform memory analysis on the same to figure the top consumers. I was able to do the same with V1 tool, but in V2 theres a hard restriction of 10000 keys.

I have an M1 mac so theres no option of using V1 tool, can you please suggest alternatives for performing the above analysis?

Thanks, Mohan

[GnaneshKunal]

Hey @mohan-venkatakrishnan,

Can you tell us what is the issue you are facing with V1 on M1 Mac?

[mohan-venkatakrishnan]

Heyy @GnaneshKunal

Thanks for the reply.

Actually i was under the assumption that v1 is not supported for m1 mac cause it wasn't showing up in the dropdown on supported OS while downloading (please refer to the below image)

But I downloaded it anyways, and it actually seems to working fine with m1 macs. As a suggestion, we can add M1 Mac in the dropdown as well for better clarity

Regards, Mohan

[leopereshz]

In advanced settings there are a config "Keys to Scan in Browser". As it's only applied to Browser page, I need an option to set the limit in "Analysis Tools", because it still shows only 10k. My redis has 10M keys and I need the tool allows me to increase at least some more keys (100k i.e.) to see which are the keys consuming memory.

[LeoSko]

I have the same issue with analysis page. My test DB has ~600k keys, while in production this count is ~110M and I'm unable to identify the true source of excessive memory consumption that appeared lately in prod with partial scan.

[esigma5]

Hi there. I changed the "Keys to Scan in List view" setting, but then on the Browser view scan is still done by 10k elements. I am using version 2.32.0 on mac/m1.

[ViktarStarastsenka]

hi @esigma5 , the setting under the Advanced settings means another thing by default, it scans 500 indeed, but when you set filters per key name or key type, there can be cases when you cannot find 500 keys (e.g. when a pattern is rare), in this case, the value in the Advanced settings sets the cap for the number of keys to scan basically, it means “scan per 500 until you find 500 keys OR you scanned X keys (and still found less than 500)” hope this helps

[esigma5]

Hi @ViktarStarastsenka , I understand. It then would be incredibly helpful to be able to change this other value. We operate with a ~2 billion keys database, and normally it would be desirable a longer scan instead of having to press the button repeatedly so many times.

[ViktarStarastsenka]

@esigma5 , can you please describe your use case? scanning more keys may not be an ideal option, especially for the Tree view, because RedisInsight will store the keys in memory and most probably you'll get the OOM error much sooner than scanning a ~2 billion keys database. I may be able to suggest alternatives based on your use case.

[esigma5]

Hi Viktar,

We won't even try to scan 2 billion keys. But at least 100k-500k keys. It pretty much depends on the size of the keys to load. They basic idea is to avoid us to keep pressing the button "scan more" for 20/50 times.

[micbox]

Hi Viktar,

I also would like to ask you to add the option to scan a custom number of keys, since we also have a huge key DBs (from hundreds of thousands to a couple of million keys) and pressing hundreds times "Scan more" button is irrational when your data needs to be fully scanned, especially to properly display the tree view. Our team intensively used this feature in the phased out RDM which has been replaced by RedisInsight, and missing this feature seriously affects our productivity in debugging applications.

[micbox]

In reality the matter is that we are using the Tree View very actively and there are many keys and sub-keys (multi-level keys) used for data grouping. For example, each first-level key group is some data status with records grouped under this status by subkey (used for queue processing per status) and when we scan keys by small portions we can't see all the existing statuses (until all data is scanned) to explore further into the data being in these statuses. This makes very complex to support and troubleshoot issues in the system relying on this data. Please, create an option to set the keys scan size for Tree view. Other example is when we store data queues (for example bulk message batches) under username key - i.e. userName:batchId:collection - and the same issue arises: we can't see all the users now having message batch queues in the Tree view until we scan all the records. It would be very convenient to click on the userName node to see all the user's batches and further data (as it was the case in good old Redis Desktop Manager). And as you understand, the data is dynamic in this case with accidental data volume spikes when big-size batches are being processed via Redis at a moment. Because of this limitation of small-portion scanning the Tree View becomes of little or no use in this case.