Paginating Relationships

[kyleboe]

When including related resources in a request, all associated records are returned, which can lead to performance issues and excessive data transfer when dealing with larger datasets.

Example

defmodule UserController do
  plug JSONAPI.QueryParser,
    include: ~w(posts),
    view: UserView
  # . . .
end

defmodule UserView do
  def fields do
    [:name]
  end

  def links(data, conn) when not is_list(data) do
    # Some actual pagination logic
    %{
      first: "foo",
      last: "bar",
      next: "baz",
      prev: "bang"
    }
  end

  def relationships do
    [posts: PostView]
  end
end

When a request to /users/1?include=posts is made, ideally the ability to limit the number of posts returned in a given request and to paginate the nested relationship. Currently, this does not exist and because of the special treatment of the links callback, it is merged into the data key as well as the top level document.

{
  "data": {
    "type": "users",
    "id": "1",
    "attributes": {
      "name": "Foo Barton"
    },
    "links": {
        "first": "foo",
        "last": "bar",
        "next": "baz",
        "prev": "bang",
        "self": "http://www.example.com/users/1"
    },
    "relationships": {
      "posts": {
        "data": [
          {
            "id": "1",
            "type": "posts"
          },
          {
            "id": "2",
            "type": "posts"
          }
          // . . .
          // Tens of thousands of records
        ]
      }
    }
  },
  "links": {
    "first": "foo",
    "last": "bar",
    "next": "baz",
    "prev": "bang",
    "self": "http://www.example.com/users/1"
  },
  "included": []
}
    

Proposed Solution

I believe allowing for paginating nested relationships requires being less prescriptive about the usage of links. I acknowledge that pagination is outside the scope of this lib; with that in mind, I believe a change made to resolve this issue would further emphasize the line between pagination and JSONAPI serialization.

While it would be a significant change, I believe links should be removed as a responsibility of this lib. This can be handled by allowing other keys to be merged in when render is called:

defmodule UserController do
  plug JSONAPI.QueryParser,
    include: ~w(posts),
    view: UserView

  def show(conn, %{"id" => id}) do
    user = Users.get_by_id_and_load_posts(id)

    conn
    |> put_view(UserView)
    |> render("show.json",
      %{
        data: user,
        links: Paginator.paginate_posts(conn, user)
      }
    )
end

That way a request like /users/1?include=posts&page[posts][number]=1&page[posts][size]=10 could be made.

There may be a better way to handle this within the context of the current relationships definition as well.

  def relationships do
    [posts: {PostView, :paginate}]
  end

However, everything I've come up with to back the above style of configuration has only furthered the coupling between this lib and pagination, which would seem to move things in the wrong direction.

Thoughts?

[github-actions[bot]]

This issue has been automatically marked as "stale:discard". We are sorry that we haven't been able to prioritize it yet. If this issue still relevant, please leave any comment if you have any new additional information that helps to solve this issue. We encourage you to create a pull request, if you can. We are happy to help you with that.

[github-actions[bot]]

Closing this issue after a prolonged period of inactivity. If this issue is still relevant, feel free to re-open the issue. Thank you!