scrapy-rotating-proxies
scrapy-rotating-proxies copied to clipboard
Add dynamic signaling
Adding and removing proxies is now possible through signals "ADD_PROXY" and "REMOVE_PROXY".
Additionally when a proxy is marked as dead/good a signal "DEAD_PROXY"/"GOOD_PROXY" is sent.
This enables the middleware to adapt proxies and dynamically spawn/kill proxies if the user wants.
Codecov Report
Merging #38 into master will increase coverage by
1.34%
. The diff coverage is66.66%
.
@@ Coverage Diff @@
## master #38 +/- ##
==========================================
+ Coverage 51.68% 53.03% +1.34%
==========================================
Files 5 5
Lines 238 264 +26
Branches 39 43 +4
==========================================
+ Hits 123 140 +17
- Misses 109 115 +6
- Partials 6 9 +3
Impacted Files | Coverage Δ | |
---|---|---|
rotating_proxies/middlewares.py | 24.61% <0.00%> (-0.39%) |
:arrow_down: |
rotating_proxies/expire.py | 77.19% <72.00%> (-1.70%) |
:arrow_down: |
Hi! Nice PR, thanks a lot! What do you think about using constants as signals instead of strings? The way it's done in Scrapy:
# scrapy/signals.py
spider_opened = object()
spider_idle = object()
spider_closed = object()