scrapy-rotating-proxies icon indicating copy to clipboard operation
scrapy-rotating-proxies copied to clipboard

Add dynamic signaling

Open n1kn4x opened this issue 4 years ago • 2 comments

Adding and removing proxies is now possible through signals "ADD_PROXY" and "REMOVE_PROXY".

Additionally when a proxy is marked as dead/good a signal "DEAD_PROXY"/"GOOD_PROXY" is sent.

This enables the middleware to adapt proxies and dynamically spawn/kill proxies if the user wants.

n1kn4x avatar Mar 19 '20 16:03 n1kn4x

Codecov Report

Merging #38 into master will increase coverage by 1.34%. The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master      #38      +/-   ##
==========================================
+ Coverage   51.68%   53.03%   +1.34%     
==========================================
  Files           5        5              
  Lines         238      264      +26     
  Branches       39       43       +4     
==========================================
+ Hits          123      140      +17     
- Misses        109      115       +6     
- Partials        6        9       +3     
Impacted Files Coverage Δ
rotating_proxies/middlewares.py 24.61% <0.00%> (-0.39%) :arrow_down:
rotating_proxies/expire.py 77.19% <72.00%> (-1.70%) :arrow_down:

codecov-io avatar Mar 19 '20 16:03 codecov-io

Hi! Nice PR, thanks a lot! What do you think about using constants as signals instead of strings? The way it's done in Scrapy:

# scrapy/signals.py

spider_opened = object()
spider_idle = object()
spider_closed = object()

StasDeep avatar Jun 07 '20 07:06 StasDeep