scrapy-splash
scrapy-splash copied to clipboard
SplashAwareFSCacheStorage does not handle HTTP 498 requests
Hi,
I just restarted my crawler and I had HTTP 498 errors, which also in the next tries never worked.
I see the following information has been cached by SplashAwareFSCacheStorage
{"error": 498, "type": "ExpiredArguments", "description": "Arguments stored with ``save_args`` are expired", "info": {"expired": ["lua_source"]}}
It would make sense to specifically handle this case so that the next request will not lead to read this cache (or to not store it at all)