Generic Paging Support

[markcowl]

We need to support pagination in TypeSpec. There are a few goals with this proposal:

1. Support common pagination patterns out of the box, e.g. at a minimum:
   1. Client-driven pagination:
      1. Offset and limit.
      2. pages
   2. Server-driven pagination:
      1. keyset
      2. time-based
      3. RFC5988
2. Support for customizing the implementation of these patterns
3. Support bespoke patterns via some form of customization and/or low-level vocabulary.
4. No must-understand vocabulary - emitters which don't intend to support pagination should continue to work, offering an API that allows a user to manually paginate.

It is important to note that while there are some general high-level patterns in play, this list is not exhaustive (people do all kinds of things!) and that for some patterns there are variants in terms of the names of the fields, the location of the fields (header, query string, request/response body), and additional metadata that is present.

Pagination Decorators

The following decorators are proposed for the core compiler.

In order for an operation to be paginated, the following requirements must be met:

1. It must have an @list decorator applied.
2. It must have either
   1. A parameter marked with @offset or @pageIndex
   2. A response field marked with @nextLink
   3. A response header named "Link" containing a Link type with a relation type of "next"

The requirements for 2 are not exclusive - an operation might have any or all of these items, in which case it is up to clients to decide which method to use for pagination.

General pagination parameter decorators

1. @pageSize: the parameter that controls the maximum number of items to include in the page.

Client-driven pagination

Client-driven pagination is achieved by annotating the parameters of the endpoint with the following decorators:

• @offset: the number of items to skip. Clients are expected to track how many items have already been seen in order to set this properly.
• @pageIndex: the page of tiems to return. Clients are expected to track the current page in order to set this properly.

Either offset or pageIndex is required in order for clients to generate a client-driven paginated interface.

Server-driven pagination

Server-driven pagination is achieved by annotating a paginated response with the following decorators:

• @continuationToken: the token to get the next page, specified on both an operation parameter and in the response.
• @nextLink: the link to the next page.
• @prevLink: the link to the previous page.
• @firstLink: the link to the first page.
• @lastLink: the link to the last page.

Either continuationToken or nextLink must be defined in a paged response, and everything else is optional. The type of the Link fields must be url or relativeUrl. When continuationToken is provided, clients are expected to assemble the nextLink by passing the response's continuation token to the corresponding parameter of the paginated operation, and copy all other parameters as-is.

RFC 5988 Link type

RFC 5988 defines a Link which is comprised of:

• a link context (in the case of HTTP, the current page, in general, inferred from where the link is found)
• a link relation type (e.g. "next" or "previous" for pagination)
• a link target (e.g. a url or relativeUrl)
• optionally, target attributes (e.g. title="next page").

This RFC defines link relation types useful for pagination, and is commonly used for this purpose (e.g. GitHub and DigitalOcean APIs). In order to support pagination using this RFC without creating new must-understand decorators or special serialization logic, we need the following types defined:

// defined in core
/**
 * An RFC 5988 Link
 */
model Link {
  target: url;
  rel: string;
  attributes?: Record<string>
}

// defined in HTTP
/**
 * An HTTP Link header, which is a string.
 */
@Internal.LinkHeader
scalar LinkHeader<T extends Record<url> | Link[]> extends string;

extern dec LinkHeader(target: Reflection.Scalar);

Examples

Client-driven pagination

Offset and limit:

model Page<T> {
  items: T[];
}
@list op Foo(@offset @query offset: int32, @pageSize @query limit: int8): Page<T>;

Pages:

model Page<T> {
  items: T[];
}
@list op Foo(@pageIndex @query page: int32, @pageSize @query pageSize: int8): Page<T>;

Next and prev links in response envelope

model Page<T> {
  @nextLink next: url;
  @prevLink prev: url;
  items: T[];
}
@list op Foo(): Page<T>;

Or, nested in the envelope somewhere:

model Page<T> {
  links: {
     @nextLink next: url;
     @prevLink prev: url;
  };
  items: T[];
}
@list op Foo(): Page<T>;

Next and prev links in header (RFC 5988)

model Link {
  target: url;
  rel: string;
  attributes?: Record<unknown>
}

scalar LinkHeader<T extends Record<url> | Link[]> extends string;

model Page<T> {
  @header Link: LinkHeader<{
   next: url,
   previous: url,
   first: url,
   last: url
  }>;
  items: T[];
}

@list op Foo(): Page<T>;

Or, using link headers with custom attributes:

model Page<T> {
  @header Link: LinkHeader<[
    { rel: "next", target: url, attributes: { title: string } }
  ]>;
  items: T[];
}

@list op Foo(): Page<T>;

[daviwil]

@johanste will discuss this with the API Review Board and report back.

[connorjs]

Hello 👋🏻 - I found this issue via the TypeSpec docs. Are there any updates in the pagination space?

For context, I plan to build paginated APIs in the coming week(s), so I’m asking in advance. TypeSpec may already fully support what I need though.

[markcowl]

@bterlson please use this as the imp issue for generic paging support

[markcowl]

est: 21 pri: 1

[timotheeguerin]

Starting implementation and finding a few issues:

1. Does it actually make sense for all the @prevLink, @nextLink, etc. decorators to be in core vs in Http
   • If not then how do we make core understand there is paging there as well or is it seens as enhanced paging from http
2. SHould there be a decorators to describe where is the items property.