Add paging to API query responses

[benjaminion]

Feature request from a prominent user of Teku.

For example, with the /eth/v1/beacon/states/{state_id}/validators endpoint.

Is it possible to add pagination to this call to not receive all 300K+ validators at once.

Example: /validators?pagination.limit=1000&pagination.offset=1000

[rolfyone]

could potentially support a syntax like 1..10 etc..

[araa47]

Seeing issues with Teku clients timing out when querying ceratin slots ,

this has been happening with slots after June 23rd

Example Slot

"eth/v1/beacon/states/4139392/validators"

Ive tried passing in the extra parameter status but this still has the same issue:

Simple Script to Re-Create the load

import requests
import time 

url = "node_url"
valitador_statuses = ['pending_initialized', 'pending_queued', 'active_ongoing', 'active_exiting', 'active_slashed', 'exited_unslashed', 'exited_slashed', 'withdrawal_possible', 'withdrawal_done']
slot = 4139392
base_url = "eth/v1/beacon/states/4139392/validators"

for status in valitador_statuses:
    resp = requests.get(f"https://{url}/{base_url}", params={"status": status})
    print(resp.status_code)
    print(resp.elapsed.total_seconds())
    time.sleep(1)

[rolfyone]

@benjaminion do we have more context around the actual problem being experienced? Pagination is a potential solution, but more context on the problem scope we're solving would be helpful...

If the problem being experienced is timeouts, then its likely pagination may not actually help a lot, as most timeouts are likely to be on state regeneration, rather than on encoding (node falling behind would be another potential)

@araa47 its possible / likely that your timeouts are due to state regeneration, and load like your script will definitely exacerbate the problem, you may find that for a few slots it times out, then it starts getting really quick again for a period of time. Every 2048 states are stored, which means that for the other 2047, they need to be regenerated, and as you get further from the stored state, it will slow down. To check the database version, can look at the db.version file in the beacon folder... Currently we have no real way of showing the exact slots of finalized states in storage... It's possible to increase the frequency of stored finalized states by using --data-storage-archive-frequency if you're using an archive database other than leveldb-tree, which would mean you need to regenerate less states to get to the data you need...

Finalized state access is definitely more expensive than non-finalized state access, and potentially if there are specific things that are often required, we could do things to optimise accessing that specific data somehow, but it really depends what the problem we're trying to solve is...

[ajsutton]

Going to close this one - paging isn't part of the standard REST API and we do support streaming the response as it's generated now which will help significantly with response times and avoiding timeouts (but we still have to regenerate the state which is usually the slowest part - pagination won't help with that though).