Pagination of included collections in compound documents

[ghostal]

I'm unsure if the feature I'm describing here is not implemented, or if I am just not configuring things correctly to make it work this way.

If I have a Post model which HasMany Comment models, and I include=comments when requesting from /api/v1/posts/1, I see something like this (note 15 comments):

{
  "jsonapi": {
    "version": "1.0"
  },
  "links": {
    "self": "https://localhost/api/v1/posts/1"
  },
  "data": {
    "type": "posts",
    "id": "1",
    "attributes": {
      "id": "1",
      "name": "Post 56759",
      "uuid": "1243f3d6-3e2d-4ee7-b58d-20055d3b9d2b",
      "created_at": "2022-03-21T12:00:04.000000Z",
      "updated_at": "2022-03-21T12:00:04.000000Z",
      "is_new": false
    },
    "relationships": {
      "comments": {
        "links": {
          "related": "https://localhost/api/v1/posts/1/comments",
          "self": "https://localhost/api/v1/posts/1/relationships/comments",
        },
        "data": [
          {
            "type": "comments",
            "id": "1"
          },
          {
            "type": "comments",
            "id": "2"
          },
          {
            "type": "comments",
            "id": "3"
          },
          {
            "type": "comments",
            "id": "4"
          },
          {
            "type": "comments",
            "id": "5"
          },
          {
            "type": "comments",
            "id": "6"
          },
          {
            "type": "comments",
            "id": "7"
          },
          {
            "type": "comments",
            "id": "8"
          },
          {
            "type": "comments",
            "id": "9"
          },
          {
            "type": "comments",
            "id": "10"
          },
          {
            "type": "comments",
            "id": "11"
          },
          {
            "type": "comments",
            "id": "12"
          },
          {
            "type": "comments",
            "id": "13"
          },
          {
            "type": "comments",
            "id": "14"
          },
          {
            "type": "comments",
            "id": "15"
          }
        ]
      }
    },
    "links": {
      "self": "https://localhost/api/v1/posts/1"
    }
  },
  "included": [
    // ...
  ]
}

But I'd prefer to be able to see something like this (note 5 included Comment models, and additional relationships.comments.links and relationships.comments.meta fields showing the pagination options):

{
  "jsonapi": {
    "version": "1.0"
  },
  "links": {
    "self": "https://localhost/api/v1/posts/1"
  },
  "data": {
    "type": "posts",
    "id": "1",
    "attributes": {
      "id": "1",
      "name": "Post 56759",
      "uuid": "1243f3d6-3e2d-4ee7-b58d-20055d3b9d2b",
      "created_at": "2022-03-21T12:00:04.000000Z",
      "updated_at": "2022-03-21T12:00:04.000000Z",
      "is_new": false
    },
    "relationships": {
      "comments": {
        "links": {
          "related": "https://localhost/api/v1/posts/1/comments",
          "self": "https://localhost/api/v1/posts/1/relationships/comments",
          "first": "https://localhost/api/v1/posts/1/comments?page%5Bnumber%5D=1&page%5Bsize%5D=5",
          "last": "https://localhost/api/v1/posts/1/comments?page%5Bnumber%5D=3&page%5Bsize%5D=5",
          "next": "https://localhost/api/v1/posts/1/comments?page%5Bnumber%5D=2&page%5Bsize%5D=5",
        },
        "meta": {
            "page": {
                "currentPage": 1,
                "from": 1,
                "lastPage": 3,
                "perPage": 5,
                "to": 5,
                "total": 15
            }
        },
        "data": [
          {
            "type": "comments",
            "id": "1"
          },
          {
            "type": "comments",
            "id": "2"
          },
          {
            "type": "comments",
            "id": "3"
          },
          {
            "type": "comments",
            "id": "4"
          },
          {
            "type": "comments",
            "id": "5"
          }
        ]
      }
    },
    "links": {
      "self": "https://localhost/api/v1/posts/1"
    }
  },
  "included": [
    // ...
  ]
}

This would mean fetching a collection of 15 Post models with potentially many Comment models each still only returns a maximum of 75 Comment models in the compound document (15 Post models × max 5 Comment models each).

My reading of the spec leads me to believe this is possible within the constraints of the specification, but I'm not sure if/how this package supports it.

[BenWalters]

Hey @ghostal As per slack: Since you can have unlimited comments on a post, a better solution would probably be just to use the relationship endpoint /posts/{id}/comments and this can be paginated. See https://laraveljsonapi.io/docs/1.0/schemas/eager-loading.html#disabling-on-specific-relations

[lindyhopchris]

Yes so pagination on relationship endpoints is fully supported.

It is not supported within included relationships in a compound document, i.e. you cannot paginate comments when retrieving a posts model, as per your original message. This is because include paths are the equivalent of model eager loading. As per the Laravel docs:

The limit and take query builder methods may not be used when constraining eager loads.

I.e. you cannot use pagination to constrain eager loading.

[ghostal]

Thanks for the responses guys.

@lindyhopchris, what are the chances this might be supported in future? I'm guessing quite slim, as the package mostly leans on Eloquent's eager loading strategy to bring child objects in without N+1 problems.

It did occurr to me though that it would still be possible in certain situations to constrain eager loads using where clauses, for example if child models had a rank/sort ordering column, to achieve a form of pagination, but it definitely feels like a "hack", and obviously wouldn't won't work in all (or even most) situations...

[lindyhopchris]

No plans to support this at the moment, though see below for a lot more detail on this.

In all the production APIs I've built with the JSON:API spec, we've just been consistent with relationships that could return 0 to 1000s (in theory) related models - they've always been links-only on the primary resource, and then must be accessed via the relationship endpoint. Typically we'd force pagination so that there's no risk of the client requesting so many resources (e.g. the entire relationship) that it could break the API.

Could this be supported in the future? The answer is a partial yes.

It could never be supported on this endpoint:

GET /api/v1/posts i.e. lots of posts.

Because the only way to paginate the comments of every post returned would result in an N+1 scenario. So that's never going to be supported.

In theory however it could be supported on this endpoint:

GET /api/v1/posts/123 i.e. one post.

That would work for relationships on the primary resource but would not work at depth. I.e. you can paginate the comments relationship of the primary post, but you couldn't paginate the include path comments.responses.

This would work without causing N+1 problems, because you are only loading a specific post's comments, rather than the comments of lots of posts, so you don't get an N+1 problem. However, it doesn't work for the comments.responses include path because then you would get the N+1 problem.

I'll be honest it's only just now that I've twigged that this is possible to partially support. I haven't yet had any thoughts about how it could be implemented. The eager loading implementation is one of the most complicated bits of the laravel-json-api/eloquent package. I did have a plan to refactor the eager loading implementation to tidy it up, as it got pretty messy as I worked through all the things it had to cover off. So it's possible if I was refactoring it I could look to introduce support for paginating relationships on a primary resource.

However, I couldn't commit to any timelines as my open source time is limited at the moment.