scrapy-splash icon indicating copy to clipboard operation
scrapy-splash copied to clipboard

allow SPLASH_SLOT_POLICY setting to take a path

Open iAnanich opened this issue 6 years ago • 1 comments

As mentioned under #157 in this comment: this feature must be backward compatible.

iAnanich avatar Jan 19 '18 20:01 iAnanich

I have an idea to create slot_policy.py module with variables equivalent to middleware.SlotPolicy attributes and refactor SlotPolicy's attributes to reference slot_policy.py's variables.

To get slot policy by path middleware.SplashMiddleware._set_download_slot method can be refactored to first search slot_policy argument value in known policies and then attempt to import object using scrapy.utils.misc.load_object function.

Is it better to use if-elif-else for known policies and then make an attempt to import, or if given slot_policy is unknown make attempt to import and then if-elif-else over known slot policies?

Or maybe it's better to extend load_object function to resolve attributes to, for example by adding : character before the name of the attribute. In our case 'per_domain' slot policy setting can be set as 'scrapy_splash.middleware.SlopPolicy:PER_DOMAIN' string.

New tests in test_middleware.py must be created to test different SPLASH_SLOT_POLICY values.

The new feature must be mentioned in README.rst.

  • [ ] refactor _set_download_slot
  • [ ] test different setting's values
  • [ ] update README.rst

iAnanich avatar Jan 19 '18 20:01 iAnanich