scrapy-splash

scrapy-splash copied to clipboard

$ python -V Python 3.9.0 [scrapy.core.scraper] ERROR: Error downloading

jicvi

Switch engines from safari to chromium

2

I see that the standalone splash has support for chromium choosing the engine through the args supplied to splash. How do I switch engines with scrapy splash?

veracato

Scrapy-Splash not rendering this site

2

Javascript not rendering at https://iwilltravelagain.com/latin-america-caribbean/?page=1 Can you tell me what could be the problem? my settings: ``` SPLASH_URL = 'http://localhost:8050' DOWNLOADER_MIDDLEWARES = { 'scrapy_splash.SplashCookiesMiddleware': 723, 'scrapy_splash.SplashMiddleware': 725, 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810, }...

MarkBorodin

Hello , I'm facing a problem in my spider for 'DEPTH_LIMIT' not work. ``` # setting.py # depth to crawler #-------------------------------------------------- DEPTH_LIMIT = 5 DEPTH_STATS_VERBOSE = True DEPTH_PRIORITY = 1...

kevinzhangcode

add SplashHtmlResponse (extended #115)

7

Contains rebased @atultherejput commits from #115 with more test cases and little improvements.

iAnanich

Hi everyone I am having some truoble storing the cookies after the authentication. I managed to authenticate and here is a dummy user account to play with. What I want...

federicoscabbia

How to implement Scrapy Splash in Virtual Machine

1

How do I run scrapy splash on a virtual machine with linux? Essentially, I have a lua script that requires me to send keys onto a site to log in...

lime-n

``` function main(splash) splash:set_user_agent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36') splash:go{splash.args.url,headers=splash.args.headers,} assert(splash:wait{60}) return { png = splash:png(), } end ``` website link https://www.oscaro.es I have...

maxwill-max

I am a big fan of scrapy splash. I am using scrapy with splash downloader option, but unfortunately its **unable to render** the following website built with Javascript. https://www.cogniertechnology.com/ I...

kiran-chinnapa

scrapy-splash recursive crawl using CrawlSpider not working

36

Hi ! I have integrated scrapy-splash in my CrawlSpider process_request in rules like this: ``` def process_request(self,request): request.meta['splash']={ 'args': { # set rendering arguments here 'html': 1, } } return...

dijadev