scrapy-redis icon indicating copy to clipboard operation
scrapy-redis copied to clipboard

Published 20 hours ago • verified publisher icon rmax

Redis data persistence

Open MRPTY1 opened this issue 3 years ago • 4 comments

I hope to add a function of persistence of fingerprint in scrapy-redis after the end of the crawl.

MRPTY1 avatar Apr 11 '22 08:04 MRPTY1

a function of persistence of fingerprint in scrapy-redis

set SCHEDULER_PERSIST = True in scrapy settings.py , It will works.

songhao8080 avatar Apr 13 '22 01:04 songhao8080

I prefer to store in a database or file,there are many data in Redis, but I don't want to delete them. If you read the fingerprint file before running, then store it again, this is very convenient.

MRPTY1 avatar Apr 13 '22 08:04 MRPTY1

@MRPTY1 Sorry for late reply. Pull Requests are welcome! Or we will schedule it to our TODO list.

LuckyPigeon avatar May 16 '22 07:05 LuckyPigeon

IMO, this more like a documentation feature. We could add exmaple relate to this. PRs are welcome! For now you can add correspond data persistence code in close_spider of your item pipeline.

LuckyPigeon avatar Sep 17 '22 11:09 LuckyPigeon