scrapy-selenium
scrapy-selenium copied to clipboard
Allow passing experimental_options to webdriver
This allows us to pass "experimental options" from the scrapy configuration to the webdriver. For example, when using chrome, the following settings lets us avoid downloading any images -- reducing the downloaded bytes significantly:
{
'SELENIUM_DRIVER_NAME': 'chrome',
'SELENIUM_DRIVER_EXECUTABLE_PATH': which('chromedriver'),
'SELENIUM_DRIVER_ARGUMENTS': ['--headless'],
'SELENIUM_DRIVER_EXPERIMENTAL_OPTIONS': {
'prefs': {
'profile.managed_default_content_settings.images': 2
}
},
'DOWNLOADER_MIDDLEWARES': {
'scrapy_selenium.SeleniumMiddleware': 800
}
}
PS This also addresses some problems in setup.py which, it seems to me, unnecessarily relied on pip internals.