pushshift
Requesting next 100 records
What is the best way to to continue your search from where you left off (i.e. 100 records on, like for pagination)?
I know you can request by date but this would leave gaps. I think the native API allows you to start at an ID where it will continue from there?
It's probably inefficient but I've started by querying by hour and then exponentially increasing the span when no results come back. Then when results come back I rest the span to 1 hour.
+1 Is there any better way to paginate, other than requesting by hours?
Update; I found this comment which is a good idea. I'm also trying the psaw library which seems to handle all of this under the hood.
I paginate results by taking records before the earliest timestamp of the previous results - e.g. this in the query string:
...&size=100&before=1676374987