scrapy-selenium

scrapy-selenium copied to clipboard

The canonical way to share info between `Request` and download middleware is `Request.meta`. Custom attributes `Request.custom_attr` should be avoided, cause it may be dropped after possible serialization and de-serialization in...

laggardkernel

How to do concurrent scraping?

2

I currently use single thread scraper to crawl google.com. But I have tons of search terms. scrapy-selenium only open 1 browser, so I could only search one term at one...

vbuterin2

Add a callback cb_selenium

2

This patch introduces 2 news parameters `cb_selenium` and `cb_selenium_kwargs`. The purpose of the selenium callback is to init the webpage like the scraper wants. In this callback you can use...

nfk

navigator.webdriver avoidance.

1

Hi! First of all, thanks for your amazing job! I need to scrape a site that "doesn't work" with `scrapy-selenium`. After a bit of headache, I've spotted the problem. It...

Anandir

Passing URL to parse in Scrapy Spider URL is captured using Scrapy-Selenium

7

I am trying to scrape a website which has some dropdowns, So I planned to use Scrapy Framework with Scrapy-Selenium(more here) to click around the dropdowns(Nested For loop) and then...

Mathoholic

Project Future Plan?

3

It's been a year since the last commit. This repo need maintainers.

psdon

Keep browser window open after scraping?

1

How can one keep the browser window open after scraping has finished (or aborted)? Thanks!

wondering639

Prevent kwargs override on replace

1

## Background In `scrapy`'s `Request` class, the following function is defined - ```python def replace(self, *args, **kwargs): """Create a new Request with the same attributes except for those given new...

sushinoya

#90 Support safari

Weilet

## Exception: ModuleNotFoundError: No module named 'selenium.webdriver.safari.options' ## Reason: Safari has no options module ## Solution: Not to import options module if the driver name is safari

Weilet