scrapy-rotating-proxies icon indicating copy to clipboard operation
scrapy-rotating-proxies copied to clipboard

Add Support for Priority Proxies

Open iamashks opened this issue 5 years ago • 1 comments

Hi,

You created a wonderful middleware for Scrapy. It's very useful.

I added a small feature: support for priority proxies. Using this feature, one can add two sets of proxies and give weights to them to use them together. For example, a 0.5 weight says 50% chance of using proxies and 50% chance of using priority_proxies.

I find it useful when I need to use free and premium proxies OR proxies and Tor together. I hope other users may find it useful as well. So, I'm creating a PR.

Kindly review my pull request and provide feedback.

Thank you, Ash

iamashks avatar Jan 27 '20 12:01 iamashks

Codecov Report

Merging #35 into master will decrease coverage by 5.83%. The diff coverage is 9.61%.

@@            Coverage Diff             @@
##           master      #35      +/-   ##
==========================================
- Coverage   51.68%   45.84%   -5.84%     
==========================================
  Files           5        5              
  Lines         238      277      +39     
  Branches       39       47       +8     
==========================================
+ Hits          123      127       +4     
- Misses        109      144      +35     
  Partials        6        6
Impacted Files Coverage Δ
rotating_proxies/expire.py 79.34% <100%> (+0.45%) :arrow_up:
rotating_proxies/middlewares.py 20.6% <6%> (-4.4%) :arrow_down:

codecov-io avatar Jan 27 '20 12:01 codecov-io