twitter-archive-parser
twitter-archive-parser copied to clipboard
timhutton
added randomizer to the request delay
Hello! I just added a simple randomization of the delay of request to minimize the possibility of cutoffs even more
Hello. Thanks for making a PR.
One change per PR please.
What's the motivation for the randomizing? Have we seen any problems with the fixed sleep?
For good contributions to a project I suggest:
- Make an issue first, identifying a problem or suggesting a change.
- Get consensus from the community that it's a valid issue and worth working on.
- In that issue, propose a solution, invite comments on how it would work.
- If there's broad agreement on the shape of the solution then go ahead and make a PR. If in doubt just ask.
Thank you for the help I'm learning to take this into account in the future
And as far as having a fixed delay on the request Some websites can block access to prevent web scraping, that can be easily detected if your Python script is sending multiple requests in a short period of time via the same duration. To not get banned adding random delays between queries increases the chance your hitting the sweet spot of the time delay the server will not block in at least some of your requests. I didn't make the delay too much larger than the original but it certainly doesn't hurt!
@MarquisTheCoder I'm closing this because the sleep strategy in download_better_images.py has changed completely. The other refactorings might be worth making a new PR for if you are motivated.