UUI icon indicating copy to clipboard operation
UUI copied to clipboard

[DataSources] GraphQL cursors support

Open jakobz opened this issue 2 years ago • 0 comments

Description

LazyDataSources is designed to load lists of items on demand. We use from/count in requests to handle this.

GraphQL suggests a better approach: cursors, read more here: https://graphql.org/learn/pagination/

Server can store an item index in cursor, getting the same as with from/count.

However, server can store values of fields, on which the list is sorted. E.g. ["John Doe", 123] - if list is sorted by name and Id. Then, server can do the following SQL:

SELECT.... FROM Users
WHERE name > "John Doe" or (name = "John Doe" and ID > 123)
ORDER BY name, id

This would allow using DB index to skip some records, while the usual TAKE N/LIMIT N approach would require re-scanning all records, which were already passed to the client in previous requests.

Also, this approach works better if underlying list changes. E.g., items being added on top, or removed, while user scrolls and loads more items. With from/count approach, we can get the same item twice, or skip an item. Cursors helps to avoid such problems. Note that cursors doesn't completely solve this - we still can get duplicates, when an item was moved down the list.

I suggest supporting this approach, and even to recommend using it as default - even in plain REST APIs.

Describe the solution you'd like

To support it, we can allow returning a new cursor: any field in the LazyDataSourceApiResponse. This cursor should be stored, and passed back in the API request, in the LazyDataSourceApiRequest argument, when the next part of this particular list needs to be loaded.

Let's make sure we follow GraphQL specs (https://relay.dev/graphql/connections.htm) in how cursors behave.

Let's also rework at least some of our demo API to this approach, to show it in demos and examples.

jakobz avatar Feb 24 '23 12:02 jakobz