scrapy-splash

scrapy-splash copied to clipboard

I created a spider with scrapy_splash, I hardcoded 3 urls in start_requests. When I run with any one url it is working fine for all the urls. when I put...

Pechi77

Hi, I just restarted my crawler and I had HTTP 498 errors, which also in the next tries never worked. I see the following information has been cached by SplashAwareFSCacheStorage...

tmaier

```lua function main(splash, args) assert(splash:go(args.url)) assert(splash:wait(0.5)) local performance = splash:jsfunc([[ function () { var divs = JSON.stringify(window.performance.timing); return performance; } ]]) return { title = performance(), har = splash:har(), }...

songhao8080

Hello. I'm using Splash and scrapy-splash. It's quite slow, but it works. My HTTPCACHE_DIR is filled with lot of stuff. I have HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage' as specified in the doc....

orzel

Splash response.follow not working well

1

When I use response.follow, callback response.url is not correct. It makes force you to use SplashRequest and join url to follow link. Since SplashRequest extends Request, but the follow method...

deverhjj

Make certificates from original web request response available in SplashRequest

1

I'd like to be able to include the status of whether the response to the URL being scraped used SSL or not. The challenge is that inside the `parse` method...

ned2

Will there be a new PyPI release?

1

Hello, I was wondering if we can expect a new release published to PyPI at some point? I can see a number of issues have been fixed in master, however...

nirvana-msu

diegous

This plugin only supports proxies directly via splash [1]. Using the native scrapy proxy support [2] leads to an attempt to contact splash via the configured proxy. It would be...

3hhh

Hi all, I'm quite new on splash and currently try to scrape web pages that require login. I use splash because it can render javascript which is used by the...

roberthng