scrapy-selenium

scrapy-selenium copied to clipboard

Feel many people will require synchronous solution to requests prior to response parsing to ensure best compatibility with scrapy design which keeps response parsing and request asynchronous. This utilizes the...

yash-fn

Remote webdriver not working

5

I have a docker container running selenium-chrome (the "standalone-chrome" official container) and I'm trying to get scrapy-selenium to work with it. I have this in settings.py: DOWNLOADER_MIDDLEWARES = { 'scrapy_selenium.SeleniumMiddleware':...

nareto

miaopeng233

Not sure what is going on but response.request.meta['driver'].get_screenshot_as_png() has a different result comparing with response.meta['screenshot']. I noticed while handling multiple URLs that the response.request.meta['driver'] does not have the expected webpage...

fabiobatalha

I am trying to run scrapy-selenium in an wsl pipenv environment i have added the ``` SELENIUM_DRIVER_NAME = 'chrome' SELENIUM_DRIVER_EXECUTABLE_PATH = '/usr/bin/chromedriver' SELENIUM_DRIVER_ARGUMENTS=['--headless'] ``` and ``` DOWNLOADER_MIDDLEWARES = { 'scrapy_selenium.SeleniumMiddleware':...

Mehss

Allow passing experimental_options to webdriver

1

This allows us to pass "experimental options" from the scrapy configuration to the webdriver. For example, when using chrome, the following settings lets us avoid downloading any images -- reducing...

erikkemperman

Clarify what happens when additional driver commands are executed

1

Let's say I got back my response, now I want to use the driver to do some additional things like clicking buttons or scrolling etc. ```python def get_table(self, response): driver...

antonky

Handle timeout exception from selenium and still return the page

3

Hi @clemfromspace I'm using the `wait_time` and `wait_until` to wait for a page to be rendered but, sometimes, the page renders a way I'm not expecting. If I don't use...

michelts

I just wanted to add the missing spaces :)

wujood

`Settings.get()` default value is `None`. The default value for `SELENIUM_DRIVER_ARGUMENTS` should be an empty list to support `for` loop. ```python class SeleniumMiddleware: def __init__(self, driver_name, driver_executable_path, browser_executable_path, command_executor, driver_arguments): ......

laggardkernel