Scraper not getting all posts from a group

[minimaxir]

Received multiple comments from people hitting this. Investigating.

[josesho]

I suspect its a Graph API issue and not your code. What I've found useful is to scrape posts in hourly windows; this takes a lot longer but seems to work better?

[PhanDuc]

@josesho , How to make scrape by specified hourly ? Can you write more about that ?

[lukaskawerau]

This is also happening for posts on pages, I get only about 300 back. Any ideas on how to fix this? @josesho does your approach also work for pages? How do you do it?

[minimaxir]

Looks like the API endpoint stops returning posts after 500 entries (on Pages atleast), even on older API versions. (specifically, there is no next url, and using the paging_token to simulate the next URL will return an empty set)

This won't break the scrapers which want recent information, but it will break the ones which want to gather all data from a group. That is unfortunate, but there is likely no workaround.

I will update the README where appropriate.

[lukaskawerau]

That is very unfortunate, especially because it's never mentioned in the documentation, but oh well.

[lukaskawerau]

Apparently this was a bug in the API, should work again now.

[minimaxir]

Yes, no longer seeing the behavior above, but the alternate behavior could be a portent of things to come.