scrapy-rotating-proxies
scrapy-rotating-proxies copied to clipboard
Add Support for Priority Proxies
Hi,
You created a wonderful middleware for Scrapy. It's very useful.
I added a small feature: support for priority proxies. Using this feature, one can add two sets of proxies and give weights to them to use them together. For example, a 0.5 weight says 50% chance of using proxies and 50% chance of using priority_proxies.
I find it useful when I need to use free and premium proxies OR proxies and Tor together. I hope other users may find it useful as well. So, I'm creating a PR.
Kindly review my pull request and provide feedback.
Thank you, Ash
Codecov Report
Merging #35 into master will decrease coverage by
5.83%
. The diff coverage is9.61%
.
@@ Coverage Diff @@
## master #35 +/- ##
==========================================
- Coverage 51.68% 45.84% -5.84%
==========================================
Files 5 5
Lines 238 277 +39
Branches 39 47 +8
==========================================
+ Hits 123 127 +4
- Misses 109 144 +35
Partials 6 6
Impacted Files | Coverage Δ | |
---|---|---|
rotating_proxies/expire.py | 79.34% <100%> (+0.45%) |
:arrow_up: |
rotating_proxies/middlewares.py | 20.6% <6%> (-4.4%) |
:arrow_down: |