python-sitemap icon indicating copy to clipboard operation
python-sitemap copied to clipboard

Published 20 hours ago • verified publisher icon c4software

Added a rate limiter for load reduction on the website

Open Bash- opened this issue 6 years ago • 7 comments
trafficstars

We found that websites found the scraper too resource consuming. Therefore I added this configurable rate limiter, to be able to decrease the number of requests per time period.

Bash- avatar Dec 13 '18 14:12 Bash-

Hi,

It's a nice addition, however ratelimit seems not present in the default library. It's an additional library (https://pypi.org/project/ratelimit/) ?

c4software avatar Dec 13 '18 15:12 c4software

Hi,

It's a nice addition, however ratelimit seems not present in the default library. It's an additional library (https://pypi.org/project/ratelimit/) ?

Thank you. Yes that is indeed the library which is needed

Bash- avatar Dec 13 '18 16:12 Bash-

Coincidentally, I'm adding a requirements.txt file in this PR. If the decision is made that that is desirable and the PR is merged, then you could add a line to that file:

ratelimit==2.2.1

Without this, I reckon this PR can't be merged, otherwise, it won't work for people who don't happen to have ratelimite pre-installed.

Garrett-R avatar Jun 27 '20 13:06 Garrett-R

@Garrett-R I think it still would be a nice addition @c4software Shall we include the ratelimiter?

Bash- avatar Jun 27 '20 14:06 Bash-

Hi,

Ratelimiting is a good addition, but i'm not a big fan of a tierce library. Not because its a tierce library, but I really like the idea of a « bare metal » tool.

What do you think @Garrett-R @Bash- having to rely to a tierce library is not a problem for you ?

c4software avatar Jun 29 '20 14:06 c4software

Yeah, it's an interesting point and wasn't sure what you'd think of introducing a requirements.txt file. There's definitely some benefit to having no dependencies. Going from 0 to 1 dependencies is a big difference (while going from 1 to 2 is not). I think it's really up to you and your vision for this already successful project.

A couple factors I'd be weighing if I were you:

  • Watch out for "boiling the frog". Given it currently has 0 dependencies, it might seem adding the first one is not worth it for that particular change, but if that decision is made multiple times, it may be that all the changes in aggregate are worth adding dependencies.
  • A bit harder to build the repo from source and use / contribute (favors not adding dependencies)
  • How many more dependencies might we benefit from? If these are kind of the last 2, then we have an idea of total benefits of adding dependencies. On the other hand, if we think there could be more in the future, it'll make those contributions easier. I don't have a good sense here
  • How hard is it to do without the dependencies (definitely seems doable to make both this change and my change without BTW)

On the point about how tough the package is to use for folks, I actually think we should put this on PYPI (made issue here). In that case folks would just have to do:

pip install python-sitemap

and the extra dependencies will automatically be handled.

I'd probably cast my vote for having dependencies, but I can definitely see benefits to both approaches, so I support whatever you think is best.

Garrett-R avatar Jul 03 '20 15:07 Garrett-R

You know, in light of the xz backdoor, I have a new appreciation for avoiding dependencies...

Maybe close this one?

Garrett-R avatar Apr 09 '24 22:04 Garrett-R