Add ability to cache via redis or memcache

[patcon]

Any preference? My pupa scraper is running on heroku, so filesystem-based caches are non-ideal :)

Would you accept a PR for one of these options?

[jamesturk]

if you're running in production it might be best to disable as you likely won't use the cache as it isn't intended for actual HTTP caching but development. but if you want it, either is fine by me! much appreciated

On Aug 8, 2016 6:10 PM, "Patrick Connolly" [email protected] wrote:

Any preference? My pupa scraper is running on heroku, so filesystem-based caches are non-ideal :)

Would you accept a PR for one of these options?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jamesturk/scrapelib/issues/34, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAfYvMc90WBO_1rqcQNBmqCVTUgiSDkks5qd6lAgaJpZM4Jfhth .

[patcon]

Thanks! I see what you're saying, but the city of toronto started throttling the heck out of my scrapers, so I'm trying to sort out how to not restart each time I figure out the limit ;)