Find next 10k record

[noonenene]

Hi,

I am using JsonModel.

I apologize for posting this noob question here. I am really stuck.

cust = Customer.find(Customer.plan == "plan1").all() return only 10k. How do i get the next 10k?

[sav-norem]

Hi! I believe there are currently a few ways you can do this depending on your data / needs. If you use all you can add a batch_size cust = Customer.find(Customer.plan == "plan1").all(batch_size=1000000)

You can use an offset to actually get the next 10k cust = Customer.find(Customer.plan == "plan1", offset=10000).all()

You can also change the page size and limit. In the source code, unless those options are specified they use a default page size of 1,000 - I think even with the .all cust = Customer.find(Customer.plan == "plan1", limit=1000000,page_size=1000000).all() to get 1000000 results at once.

There is a page function but my understanding is that it will only give you a single "page" of results where you specify the offset and how many results you want. But I believe you could use it similarly to the all cust = Customer.find(Customer.plan == "plan1").page(offset=0,limit=1000000)

That's a relatively new function and @dvora-h might be able to correct my understanding or add clarity.

[noonenene]

Thank you for replaying.

cust = Customer.find(Customer.plan == "plan1").all(batch_size=1000000) redis.exceptions.ResponseError: LIMIT exceeds maximum of 10000

, offset=10000 TypeError: find() got an unexpected keyword argument 'offset'

Page function not published yet to PyPi.

[banker]

@noonenene You may be hitting a RediSearch limit. See the https://redis.io/docs/stack/search/configuring/#maxsearchresults setting. You can change this by running:

FT.CONFIG SET MAXSEARCHRESULTS -1

Can you try setting this and the re-run your query?