scrapy-splash icon indicating copy to clipboard operation
scrapy-splash copied to clipboard

SplashAwareFSCacheStorage does not handle HTTP 498 requests

Open tmaier opened this issue 4 years ago • 0 comments

Hi,

I just restarted my crawler and I had HTTP 498 errors, which also in the next tries never worked.

I see the following information has been cached by SplashAwareFSCacheStorage

{"error": 498, "type": "ExpiredArguments", "description": "Arguments stored with ``save_args`` are expired", "info": {"expired": ["lua_source"]}}

It would make sense to specifically handle this case so that the next request will not lead to read this cache (or to not store it at all)

tmaier avatar Oct 18 '21 18:10 tmaier