scrapy-cluster icon indicating copy to clipboard operation
scrapy-cluster copied to clipboard

Circuit breaker design patterns

Open madisonb opened this issue 8 years ago • 0 comments

Lots of the individual components break down or crash when their required infrastructure is not available. They are dependent on kafka, redis, or zookeeper, but don't have good mechanisms always in place to handle when things go down or subsequently recover. This ticket is to ensure we both log "Unable to connect to XXXX" as well as establish any better circuit breaker design patterns to ensure the components are always up.

  • [ ] Kafka Monitor
  • [ ] Redis Monitor
  • [ ] Crawler

Note that some libraries we use may already have this in place, but would be nice to log the error while we are in a recovery state.

madisonb avatar Oct 27 '16 03:10 madisonb