scrapy-selenium icon indicating copy to clipboard operation
scrapy-selenium copied to clipboard

Allow passing experimental_options to webdriver

Open erikkemperman opened this issue 2 years ago • 1 comments

This allows us to pass "experimental options" from the scrapy configuration to the webdriver. For example, when using chrome, the following settings lets us avoid downloading any images -- reducing the downloaded bytes significantly:

{
    'SELENIUM_DRIVER_NAME': 'chrome',
    'SELENIUM_DRIVER_EXECUTABLE_PATH': which('chromedriver'),
    'SELENIUM_DRIVER_ARGUMENTS': ['--headless'],
    'SELENIUM_DRIVER_EXPERIMENTAL_OPTIONS': {
        'prefs': {
            'profile.managed_default_content_settings.images': 2
        }
    },
    'DOWNLOADER_MIDDLEWARES': {
        'scrapy_selenium.SeleniumMiddleware': 800
    }
}

erikkemperman avatar Oct 14 '21 11:10 erikkemperman

PS This also addresses some problems in setup.py which, it seems to me, unnecessarily relied on pip internals.

erikkemperman avatar Oct 14 '21 11:10 erikkemperman