feed-me icon indicating copy to clipboard operation
feed-me copied to clipboard

Ability to paginate with feeds that provide a page number instead of the next page URL.

Open chriscborg opened this issue 3 years ago • 0 comments

Description

This PR adds the ability to check the next page URL for the case where a page number is provided instead of the next page's URL. Some feeds only provide the current page information instead of the URL to the next page. Example of such feed below:

{
    "data":[
      ...
    ],
    "paging": {
        "page": 1,
        "perPage": 200,
        "pageCount": 13
    }
}

In this case, we are checking if the data provided as the next page is numeric, and a total page number is provided as well. In this case, we generate a URL using the feedUrl and appending the query string ?page={number} by incrementing the current page number. This is then repeated until the total page count provided is reached.

What is included:

  • The addition of the new field to provide the total pages
  • Edited the description in the settings form to explain that a page number can be provided instead of a URL
  • Added a database migration file to add the total page number
  • Incremented the plugin version number for migration (not sure if this is allowed)
  • Added the functionality mentioned above the to the setupPaginationUrl($array, $feed) function in the DataType class, making use of the UrlHelper function to add the query string

What is not included

  • I was not able to run the unit tests as I was getting errors while running, therefore I was not able to confirm if tests are ok, or even add any if needed. Help on this would be appreciated
  • I also was not able to test on feeds that provide a normal next page URL.

In general, help in testing would be appreciated.

Related Issues

I apologise for not creating an issue before, but I needed this feature anyway for a project I'm working on.

chriscborg avatar Mar 15 '21 17:03 chriscborg