scrapy-selenium
scrapy-selenium copied to clipboard
clemfromspace
Scrapy middleware to handle javascript pages using selenium
When the code is asynchronous, perform action by the driver from response, would cause many conflict. Add the support for perform actions before return response from middleware.
More specifically, user can configure the proxy for each of their request. If the proxy is not yet used, a new driver will be created.
Hi @clemfromspace I migrated the requirements files to setup.py and adjusted the tests to be run through `python setup.py test` (even in travis). I've also updated both test requirements versions...
Dis anyone successfully ran Scrapy-selenium with SpiderCrawl? It just never discovers new links.
Hi @clemfromspace I implemented the necessary steps to meet the issue #58. There wasn't any test of the wait_time and wait_until usage, so I added one. I decided to always...
I started using this module first it shows yellow underline in vscode under module name and when i print response.meta dict it contains only following: {'download_timeout': 180.0, 'download_slot': 'golden.com', 'download_latency':...
I was wondering if it is possible to yield requests from both standard Scrapy and scrapy-selenium in the same spider. I have mapped many requests that the website shows in...
@clemfromspace I just decided to use your package in my Scrapy project but it is just yielding normal scrapy.Requuest instead of SeleniumRequest ``` from shutil import which from scrapy_selenium import...
hey 👋 I just realized that Scrapy's USER_AGENT from settings.py doesn't equal Selenium's user_agent. I expect that `scrapy-selenium` will take that value and pass to driver's options. I solved the...
Metadata
Owner
Metadata
Scrapy middleware to handle javascript pages using selenium