Add mechanism to fetch next pages automatically

[ref-thomasledoux1]

Hi,

We often want to fetch a resource collection which contains more than 50 (default JSON API limit?) resources. At the moment getResourceCollection() gives back the first 50 resources by default, and doesn't give a way to fetch the next pages automatically. Do you think it would be possible to provide a mechanism for this? I know I can write a recursive function to handle this, but would be nice to have it built in :-) At the moment in the response there is from getResourceCollection() there is no way to know if there are more resources available (expect by checking if the returned array has length of 50, but would rather not base myself on that). Is there another call I can use where I do get the links object, so I can read the next link (if one is available)?

[shadcn]

@ref-thomasledoux1 Sorry just seeing this now.

You can get the links object using the deserialize: false option.

const response = await drupal.getResourceCollection("node--article", {
  deserialize: false,
})

response.links will have the pagination links.

[twfahey1]

Thanks, shadcn! I was able to get this going, notably had to tweak my components that were accessing the data to include .attributes, (I'm guessing due to deserialize?), but this is how I did it. Would love any feedback on this approach. I have noted the ts-errors - admittedly I haven't dug too much into those, likely an easy solution to clean that up. Notice in my case I included a "max pages" since I didn't really want all the pages. But this does seem to do what I need in grabbing several hundred items for the front end, in my case Media entities of type remote_video:

export async function getStaticProps(
  context
): Promise<GetStaticPropsResult<PageProps>> {
  // Get our menu entities.
  const menus = await getMenus(context);
  // A helper function to get the remote media entities that can be used with pagination next link.
  // If it's the first run, pass 0 as the page.
  const getNextPageOfRemoteEntities = async (page) => {
    let params = {
      include: "thumbnail"
    };
    if (page > 0) {
      params["page"] = [page];
    }

    const remote_media_entities = await drupal.getResourceCollection(
      "media--remote_video",
      {
        deserialize: false,
        params: params,
      }
    );
    return remote_media_entities;
  };
  // Check for next link and continue to get the next page of results.
  // This is a recursive function that will continue to get the next page of results until there are no more pages.
  let allRemoteMediaEntities = [];
  let finished = false;
  let page = 0;
  let pageMax = 20;
  while (!finished) {
    console.log("Getting page " + page);
    const remote_media_entities = await getNextPageOfRemoteEntities(page);
    allRemoteMediaEntities = allRemoteMediaEntities.concat(
      // Hmm... getting a ts error here, TODO: fix this.
      // @ts-ignore
    remote_media_entities.data
    );
    // Hmm... getting a ts error here, TODO: fix this.
    // @ts-ignore
    if (remote_media_entities.links.next) {
      page++;
    } else {
      finished = true;
    }
    if (page > pageMax) {
      finished = true;
    }

  }
  console.log(allRemoteMediaEntities);

  return {
    props: {
      menus,
      allRemoteMediaEntities,
    },
    revalidate: 10,
  };
}

[shadcn]

@twfahey1 We just tagged next 1.3.1 with a fix for bypassing the JSON:API page limit.

I've also published a guide here: https://next-drupal.org/guides/page-limit

In your case, let's say you want 100 media--remote_video, it would look something like this:

const remote_media_entities = await drupal.getResourceCollection("media--remote_video", {
  params: {
    "fields[media--remote_video]": "path,name", // <---- Note that `path` must be the first field.
    "page[limit]": 100, // <--- Set custom page limit here.
  }
})      

Can you update to your Next.js module 1.3.1 and test?

[twfahey1]

@shadcn Awesome... working well for me 👍🏻