scrapy-splash
scrapy-splash copied to clipboard
allow SPLASH_SLOT_POLICY setting to take a path
As mentioned under #157 in this comment: this feature must be backward compatible.
I have an idea to create slot_policy.py
module with variables equivalent to middleware.SlotPolicy
attributes and refactor SlotPolicy
's attributes to reference slot_policy.py
's variables.
To get slot policy by path middleware.SplashMiddleware._set_download_slot
method can be refactored to first search slot_policy
argument value in known policies and then attempt to import object using scrapy.utils.misc.load_object
function.
Is it better to use if-elif-else
for known policies and then make an attempt to import, or if given slot_policy
is unknown make attempt to import and then if-elif-else
over known slot policies?
Or maybe it's better to extend load_object
function to resolve attributes to, for example by adding :
character before the name of the attribute. In our case 'per_domain'
slot policy setting can be set as 'scrapy_splash.middleware.SlopPolicy:PER_DOMAIN'
string.
New tests in test_middleware.py
must be created to test different SPLASH_SLOT_POLICY
values.
The new feature must be mentioned in README.rst
.
- [ ] refactor
_set_download_slot
- [ ] test different setting's values
- [ ] update
README.rst