Add circuit breaker

[nicl]

@robertberry suggested adding a circuit breaker to the client to ensure good behaviour in the case of problems.

Let's discuss and if agreed, implement.

[robertberry-zz]

This is common practice at Netflix (see http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html for example, but there's lots of stuff on their blogs). I think building it into the client, and having it turned on by default, makes the most sense. It'd help you in situations where you have too many requests to ElasticSearch, etc., queued up, or the web boxes are saturated.

@tackley @philwills What do you guys think?

[philwills]

Yup, this seems like exactly the kind of way that having a client library can add value. Having said that it should be obvious in docs and moan loudly when it fires.

[mchv]

I am not sure what exactly a circuit breaker means on client side, but I agree the client should provide more values that just the raw API. I had 2 ideas in mind:

• 503 response: The client should be able to wait for a delay and retry before serving the error response to the user.
• Cache headers: The client should be able to cache response following cache headers and directive, exactly like a browser does.

[robertberry-zz]

This explains the concept: http://martinfowler.com/bliki/CircuitBreaker.html