scrapy-splash

scrapy-splash copied to clipboard

Cookies handling with render.html and SplashFormRequest.from_response

3

I'm using the endpoint render.html with SplashFormRequest.from_response for scraping asp.net based sites, but I can't make consecutive SplashFormRequest.from_response without losing session. I tried to set Cookies in args, meta or...

lucasginizeli

how to get redirect urls with scrapy-splash

15

I don't know how to get the redirect urls with scrapy-splash,can you help me? eg. http://xxx.xxx.xxx/1.php will redirect to http://xxx.xxx.xxx/index.php,how can I get http://xxx.xxx.xxx/index.php with scrapy-splash? Below is my code...

3xp10it

Give us more examples

1

I find this framework promising for my needs, but I'm struggling to get the correct way to implement a scrapper that clicks a button or submits a form using a...

dalepo

An example of http_method and body in splash script

4

There are examples of using cookies in the docs, but no examples of setting method and body. I think it would be useful to add it, or perhaps even add...

lopuhin

Response type still not determined correctly sometimes

1

Here is a traceback (without a way to reproduce unfortunately), which shows that most likely we still sometimes don't identify response type correctly: ``` [scrapy.core.scraper] Error downloading Traceback (most recent...

lopuhin

added SplashHtmlResponse (Fixed #114)

12

Please guide me as this is my first PR @kmike and @redapple . I had tried to fix https://github.com/scrapy/scrapy/issues/2673 by adding SplashHtmlResponse, subclassing HtmlResponse. Fixes #114, fixes #92.

atultherajput

Middleware settings for scrapy-splash with scrapy-cluster, SplashRequest not work

18

In single node scrapy project, the settings like below as your document indicate works well. ``` # ====== Splash settings ====== SPLASH_URL = 'http://localhost:8050' DOWNLOADER_MIDDLEWARES = { 'scrapy_splash.SplashCookiesMiddleware': 723, 'scrapy_splash.SplashMiddleware':...

hustshawn

Request fingerprint takes all headers and additional cookies into consideration

1

Apologies if this is stated anywhere in the documentation. From what I can see when a request object gets a chance to be run through `splash_request_fingerprint` it will have `splash.args.headers`...

hermit-crab

Bad request to Splash & HTTP status code is not handled or not allowed

4

hi kmike, i use scrapy-splash and meet a issue, when i first run 'scrapy crawl toutiao', it's run right, bug when i run it's second, it occur a issue. i...

linukey

custom user-agent

2

Hello, We realized recently that altought we had set custom user-agent in headers params of splash/args when doing a request, this header was never used by splash afterward. For now...

cp2587