Could this eventually support private subreddits & praw / authentication?

[github-userx]

[libertysoft3]

Yeah, praw support could be added somewhat easily. I think there's a 1000 posts per sub limit though https://www.reddit.com/r/redditdev/comments/8zhcmr/how_to_crawl_more_than_1000_posts_through_reddit/

Pushshift may have data for subs that went private but weren't always private.

[github-userx]

Huh? Since when does pushshift archive private subreddits? Now I am so confused. Pushshift doesn’t have any access to private subreddits AFAIK!

31. Mai 2019, 05:03 from [email protected]:

I'm pretty sure that private subs are supported now, the data comes from PushShift.

Yeah, praw could be added somewhat easily, depending on what you're trying to do with it. It would be cool to have praw fetch updates scores.

— You are receiving this because you authored the thread. Reply to this email directly, > view it on GitHub https://github.com/libertysoft3/reddit-html-archiver/issues/6?email_source=notifications&email_token=AIY4ABWE4FAMNKP776VIVHTPYCIPNA5CNFSM4HRGQO6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWUCFMY#issuecomment-497558195> , or > mute the thread https://github.com/notifications/unsubscribe-auth/AIY4ABU6KSHRX5EKWZ6NCZLPYCIPNANCNFSM4HRGQO6A> .

[libertysoft3]

Yeah I was full of it and tried to fix it with an edit. They might have some private sub data in one scenario only. Anyway what about the 1000 post limit with the reddit API?

[github-userx]

I would already be happy to get 1000 posts from a private subreddit

31. Mai 2019, 09:52 from [email protected]:

Yeah I was full of it and tried to fix it with an edit. They might have some private sub data in one scenario only. Anyway what about the 1000 post limit with the reddit API?

— You are receiving this because you authored the thread. Reply to this email directly, > view it on GitHub https://github.com/libertysoft3/reddit-html-archiver/issues/6?email_source=notifications&email_token=AIY4ABVHGEYTN6QXAQHZLPDPYDKKXA5CNFSM4HRGQO6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWUPUAI#issuecomment-497613313> , or > mute the thread https://github.com/notifications/unsubscribe-auth/AIY4ABTSRBG44GEXPZLCOGDPYDKKXANCNFSM4HRGQO6A> .

[libertysoft3]

I'm into it, pull requests accepted. Everyone that wants this please give this issue a thumbs up.

[github-userx]

Any update?

[libertysoft3]

I'm pretty busy with SaidIt lately.. I don't have this scheduled to be done currently.

[github-userx]

Ok, thanks! No worries! ;) 

[YoungerDryas89]

Hey I tried looking into adding my own implementation, but it looks like this isn't possible anymore because of Reddit's deprecation of cloudsearch. So I can't search posts by any specific date under this newer search system, and looking at r/changelog there hasn't been any updates to this newer system since then. So unless this private sub has been public before and picked up by pushift, I don't think theres anything I could do. Let me know If I missed anything or I'm totally wrong

https://www.reddit.com/r/changelog/comments/7tus5f/update_to_search_api/

[github-userx]

Thanks a lot! Really sucks how they changed the API. ;(

[libertysoft3]

Yep, PRAW talks about it in their changelog for version 6.0.0:

Removed: Subreddit.submissions as the API endpoint backing the method is no more. See https://www.reddit.com/r/changelog/comments/7tus5f/update_to_search_api/.

https://praw.readthedocs.io/en/latest/package_info/change_log.html

So I guess the best that can be done for private subs is constantly consuming new posts, and getting 1000 posts out of each different API sort that's offered.