Pagination

[metasoarous]

Is pagination of queries on your radar at all?

[comnik]

Anything involving sorting is tricky, I haven't thought much about it, and it's not at the top of the priority list right now, but here goes.

We do not want 3DF to maintain explicit order in the general case, because most input diffs (to a list of n records) can be expected to produce O(n/2) output diffs (to every connected client!). It seems a compromise would be for the backend to perform something like a top-k by the desired metric and then assign a page number to every record. Within a page, clients are responsible for presenting them in sorted order.

For pages of size k this should cut down the expected number of output diffs significantly (something like n/k * probability of any individual record moving between pages?).

How does that sound?