Crystal-Web-Archiver icon indicating copy to clipboard operation
Crystal-Web-Archiver copied to clipboard

Published 20 hours ago • verified publisher icon davidfstr

Infinitely scrolling pages: Support downloading a paginated endpoint

Open davidfstr opened this issue 2 years ago • 1 comments

If you view the page:

  • https://newsletter.pragmaticengineer.com/archive

you will notice that the page's JavaScript makes an initial request to a paginated API endpoint to get the first 12 posts:

  • https://newsletter.pragmaticengineer.com/api/v1/archive?sort=new&search=&offset=0&limit=12

If you scroll down the page slowly, you'll notice that JavaScript on the page continues to make requests to the paginated API endpoint to keep getting batches of 12 posts:

  • https://newsletter.pragmaticengineer.com/api/v1/archive?sort=new&search=&offset=12&limit=12
  • https://newsletter.pragmaticengineer.com/api/v1/archive?sort=new&search=&offset=24&limit=12
  • https://newsletter.pragmaticengineer.com/api/v1/archive?sort=new&search=&offset=36&limit=12
  • ... (etc)

Find a way to tell Crystal to fully download an infinitely scrolling page that uses a similar technique to populate items on the page.

davidfstr avatar Jun 25 '22 02:06 davidfstr