Is there a way to see consecutiveFailureCount property of Circuit Controller? [Expose circuit-breaker internal statistics]

[RichieRunner]

In order to intelligently arrive at a suitable count value for exceptionsAllowedBeforeBreaking for my CircuitBreaker policy, I want to be able to report on what the typical count of consecutiveFailureCount that the Circuit Breaker policy is tracking when it does ExecuteAsync. This would also be great for health reporting, to see how many times a circuit was broken on a day to day basis.

[reisenberger]

Hi @RichieRunner . The recommended way to capture circuit health events like how much the circuit breaks is to use the delegates on transition of state to hook in logging or push structured telemetry/metrics to your favourite sink.

exceptionsAllowedBeforeBreaking isn't exposed on the policy, but you know it yourself when configuring the policy, so you can capture it into the onBreak.

Below is some old sample code I had which sketches this - you can also run this code in dotnetfiddle here.

using System;
using Polly;
using Polly.CircuitBreaker;

public class Program
{
    const int numberOfTimesToRetry = 20;
    const int numberOfFailures = 5;
    static int tries = 0;
   
    public static void Main()
    {
		var exceptionsAllowedBeforeBreaking = 3;
		var circuitBreaker = Policy
			.Handle<InvalidOperationException>()
			.CircuitBreaker(
			    exceptionsAllowedBeforeBreaking: exceptionsAllowedBeforeBreaking,
			    durationOfBreak: TimeSpan.FromMilliseconds(100),
		        onBreak: (e, span) => Console.WriteLine("Breaking circuit for " + span.TotalMilliseconds + "ms due to " + e.Message + ". The circuit allowed " + exceptionsAllowedBeforeBreaking + " failures before breaking."),
			    onReset: () => Console.WriteLine("Trial call succeeded: circuit closing again."),
			    onHalfOpen: () => Console.WriteLine("Circuit break time elapsed.  Circuit now half open: permitting a trial call.")
		);
		var retryPolicy = Policy
			.Handle<InvalidOperationException>()
			.Or<BrokenCircuitException>()
            .WaitAndRetry(
				retryCount: numberOfTimesToRetry, 
				sleepDurationProvider: i => TimeSpan.FromMilliseconds(20), // The short timespans used here are because of timing limits imposed by .NETFiddle: https://dotnetfiddle.uservoice.com/knowledgebase/articles/282670-restrictions.  Real-world network examples would back off over seconds.
				onRetry: (e, span, i, ctx) => Console.WriteLine((e is BrokenCircuitException ? "Failed fast" : "Retrying") + " due to " + e.Message + " Waiting " + span.TotalMilliseconds + "ms first. Try " + i + " next.")
		);
		
		string result = retryPolicy.Wrap(circuitBreaker).Execute(() => GetJson());
		
		
        Console.WriteLine("Got result: " + result);
    }

    public static string GetJson()
    {
        if (tries++ <= numberOfFailures)
        {
            throw new InvalidOperationException("Something went wrong.");
        }

        return "SomeProperty: \"some interesting value\"";
    }
}

[RichieRunner]

Thanks so much for the quick response, but I think you misunderstood my question. Let's say I set my exceptionsAllowedBeforeBreaking setting at 10. If that threshold is never reached for some reason, I want to be able to report on how close I was to reaching 10. If there were 9 consecutive failures today, I want to be able to report on that 9, and how many times 9 was reached. That's the consecutiveFailureCount property that I was referring to in the source code.

[reisenberger]

Thanks @RichieRunner, that makes sense. Yes, I was focused on the second aspect of the q how many times a circuit was broken 😉

To your q: Because Polly offers two kinds of circuit-breaker (consecutive-count and advanced), implementing this is not just a simple case of add a public int ConsecutiveFailureCount to ICircuitBreakerPolicy.

We would want to tackle the general case ('Surface circuit-breaker internal stats') and do something like expose an ICircuitStatistics CircuitStatistics property on ICircuitBreakerPolicy.

Different implementations of ICircuitController<T> would then expose variants of ICircuitStatistics, such as IConsecutiveCountCircuitStatistics : ICircuitStatistics and IAdvancedCircuitStatistics : ICircuitStatistics. This approach would also slot into our longer-term goal of allowing users to extend circuit-breaker behaviour by implementing custom circuit controllers.

That would still leave some casting when consuming - something like (policy.CircuitStatistics as IConsecutiveCountCircuitStatistics).ConsecutiveFailureCount. To avoid that would probably mean extending the CircuitBreakerPolicy class-family with variants like ConsecutiveCountCircuitBreakerPolicy : CircuitBreakerPolicy and have the derived classes implement properties encapsulating the cast.

For clarity, priority-wise, for my own development time, this comes below distributed circuit breaker. We'd be happy to take a PR for the feature though, if you are interested to contribute one.

[reisenberger]

@RichieRunner If your goal is to work out how best to configure exceptionsAllowedBeforeBreaking, you may also be interested in this discussion around circuit-breaker whatiffery. An exploration strategy can be to go initially to production with an inert circuit-breaker by setting durationOfBreak: TimeSpan.Zero. Such a circuit-breaker will still cycle through the states (you can attach logging/alerts to track when breaks occur), but with TimeSpan.Zero the breaker will have no effect on your system operation, so you can experiment with finding a suitable exceptionsAllowedBeforeBreaking value before giving the circuit breaker 'teeth', if you like.

[martincostello]

@martintmk Do we have enough events in the metrics in v8 that would allow people to do this sort of thing without us having to expose the internals directly?

[martintmk]

@martintmk Do we have enough events in the metrics in v8 that would allow people to do this sort of thing without us having to expose the internals directly?

Nope, the metrics currently do not include CBs internal statistics.