scrapy-splash icon indicating copy to clipboard operation
scrapy-splash copied to clipboard

scrapy cache not used for splash requests

Open orzel opened this issue 3 years ago • 0 comments

Hello. I'm using Splash and scrapy-splash. It's quite slow, but it works. My HTTPCACHE_DIR is filled with lot of stuff. I have HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage' as specified in the doc.

Nevertheless, if i restart crawling, it's still very slow, and i can see in splash (docker logs) that all requests are done again.

So basically, my cache is filled but not used. How can I debug this ?

orzel avatar Aug 29 '21 20:08 orzel