optimade-python-tools icon indicating copy to clipboard operation
optimade-python-tools copied to clipboard

Potential optimisations for requests that do not ask for a filter

Open ml-evs opened this issue 1 year ago • 2 comments

e.g., on server startup, take a series of chunked blobs corresponding to the default sort order of the API, then if a user requests with no filter and no page limit, return this static chunked data directly.

cc @JonathanSchmidt1 @eimrek (maybe this is a useful optimisation for e.g., the Li-ion set too)

ml-evs avatar Jun 12 '24 08:06 ml-evs

thanks @ml-evs, with basic indexes in mongodb, the default load can be made fairly fast except for the total number of structures (data_returned). Might be worth adding this just for that number. Or alternatively use the mongodb estimated document count for the filterless query (but i guess this can be wrong is some very rare cases?)

eimrek avatar Jun 12 '24 11:06 eimrek

Maybe this is a good idea after all, if a web client always hits the same request, we could cache these as JSON files that are served statically rather than via the API, e.g., via the config file:

"static_index": ["?page_limit=100&response_fields=id&page_offset=0"]

ml-evs avatar Jun 13 '24 12:06 ml-evs