Redis RecordManager

[MJDeligan]

I've been experimenting with a Redis-based RecordManager for Indexing and have come up with an implementation that I would like to open a PR for.

I would like to resolve the following first:

Redis search has a configurable limit on the maximum amount of search results. See https://redis.io/docs/interact/search-and-query/basic-constructs/configuration-parameters/#maxsearchresults. Unless a limit is passed, the search only returns 10 documents, but if the limit is set higher than the max search result config it throws an error. As far as I can tell this cannot be bypassed by the offset.

This could lead to problems for users who are unaware of this limitation and might only cause problems later on, when they're scaling and hitting the limit. I've implemented a check for whether redis search couldn't return all matching documents, but they'd still only notice when it actually fails.

Currently, I'm thinking of logging a warning about this limitation, which can optionally be turned off. Do you have any other suggestions on handling this limitation (or circumventing it)?

[dosubot[bot]]

🤖

That's great to hear that you've been working on a Redis-based RecordManager and have identified an important limitation in the Redis search functionality. Your idea of logging a warning about this limitation, which can optionally be turned off, sounds like a good approach. It's important to make users aware of potential issues, especially when it comes to scaling. I encourage you to go ahead and open a PR with your implementation. Your contribution will be valuable to the project!

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.