App-vNext
[Question]: How to use a CircuitBreaker(basic or advanced) policy with multiple retry policies
What are you wanting to achieve?
I have attached my usecase or question in the attached document. Please help either in polly 7 or 8 version. Thanks in advance.
What code or approach do you have so far?
I have attached the code snippet in the attachment.
Additional context
No response
///
///
AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
.HandleResult<HttpResponseMessage>(r => _set1.Contains(r.StatusCode))
.Or<BrokenCircuitException>()
.WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetry1);
return waitAndRetryPolicy;
}
///
//jitter strategy with an ExponentialBackoff
var exponentialBackoff = Backoff.ExponentialBackoff(initialDelay: TimeSpan.FromSeconds(1), retryCount: 3);
AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
.HandleResult<HttpResponseMessage>(r => _set2.Contains(r.StatusCode))
.Or<BrokenCircuitException>()
.WaitAndRetryAsync(sleepDurations: exponentialBackoff, onRetry: OnRetry2);
return waitAndRetryPolicy;
}
///
AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
.HandleResult<HttpResponseMessage>(r => _set3.Contains(r.StatusCode))
.Or<BrokenCircuitException>()
.WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetry3);
return waitAndRetryPolicy;
}
///
})
.AddPolicyHandler(retryPolicies.WaitAndRetrySet1Async())
.AddPolicyHandler(retryPolicies.WaitAndRetrySet2Async())
.AddPolicyHandler(retryPolicies.WaitAndRetrySet3Async())
.AddPolicyHandler(retryPolicies.BasicCircuitBreakerForAllStatusCodesAsync());
My use case below:
I will submit 2 requests at a time GetEmployeeByName I expect 429 response here GetEmployeeById I expect 502 response here
In this case, GetEmployeeByName and GetEmployeeById requests will be submitted. These requests will fail as expected(intended to fail for testing purpose). Now circuit will transition to OPEN state for 10 seconds, after 10 seconds circuit will transition to Half-Open and allow either GetEmployeeByName or GetEmployeeById to process. Once all these retry requests completed. After sometime again I will submit the same requests and I should be able to see the same behaviour. But I am not able to see the desired output. May I know whether I am missing something or can you please share me any piece of example code to achieve this. Thanks in advance.
Note: Retry policies are created in one class and circuit breaker policy in another class.
@Mahantesh-DotNet Please allow me to reformat your shared code fragment to make it more legible
Retry 1
///
/// Retry these codes for 3 times for every second
///
private static readonly List _set1 =
[
HttpStatusCode.Unauthorized //401
];
///
/// Policy to retry for 3 times for every second
///
public AsyncRetryPolicy WaitAndRetrySet1Async()
{
//jitter strategy with ConstantBackoff
var constantBackoff = Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(1), retryCount: 3);
AsyncRetryPolicy < HttpResponseMessage > waitAndRetryPolicy = Policy
.HandleResult < HttpResponseMessage > (r => _set1.Contains(r.StatusCode))
.Or < BrokenCircuitException > ()
.WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetry1);
return waitAndRetryPolicy;
}
Retry 2
///
/// Retry these codes for 3 times exponentially 1,2,4
///
private static readonly List _set2 =
[
HttpStatusCode.TooManyRequests //429
];
///
/// Policy to retry for 3 times for 1,2,4 seconds
///
public AsyncRetryPolicy WaitAndRetrySet2Async()
{
//jitter strategy with an ExponentialBackoff
var exponentialBackoff = Backoff.ExponentialBackoff(initialDelay: TimeSpan.FromSeconds(1), retryCount: 3);
AsyncRetryPolicy < HttpResponseMessage > waitAndRetryPolicy = Policy
.HandleResult < HttpResponseMessage > (r => _set2.Contains(r.StatusCode))
.Or < BrokenCircuitException > ()
.WaitAndRetryAsync(sleepDurations: exponentialBackoff, onRetry: OnRetry2);
return waitAndRetryPolicy;
}
Retry 3
///
/// Retry these codes for 3 times every 5 second
///
private static readonly List _set3 =
[
HttpStatusCode.BadGateway, //502
HttpStatusCode.GatewayTimeout //504
];
///
/// Policy to retry for 3 times for every 5 second
///
public AsyncRetryPolicy WaitAndRetrySet3Async()
{
//jitter strategy with ConstantBackoff
var constantBackoff = Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(5), retryCount: 3);
AsyncRetryPolicy < HttpResponseMessage > waitAndRetryPolicy = Policy
.HandleResult < HttpResponseMessage > (r => _set3.Contains(r.StatusCode))
.Or < BrokenCircuitException > ()
.WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetry3);
return waitAndRetryPolicy;
}
Circuit Breaker
///
/// This is the circuit breaker policy to work with the above set of all status codes _set1, _set2, _set3.
/// That is a common circuit breaker policy which should work with a set of status codes.
///
public AsyncCircuitBreakerPolicy BasicCircuitBreakerForAllStatusCodesAsync()
{
AsyncCircuitBreakerPolicy basicCircuitBreakerPolicy = Policy
.HandleResult(r => _set1.Contains(r.StatusCode) || _set2.Contains(r.StatusCode) || _set3.Contains(r.StatusCode))
.CircuitBreakerAsync(handledEventsAllowedBeforeBreaking: 2,
durationOfBreak: TimeSpan.FromSeconds(10),
onBreak: OnBreak, onReset: OnReset, onHalfOpen: OnHalfOpen);
return basicCircuitBreakerPolicy;
}
I am applying all these policies to my RestApi as below
builder.Services.AddHttpClient(“TestClient”, client =>
{
client.BaseAddress = new Uri(http: //localhost:5000);
})
.AddPolicyHandler(retryPolicies.WaitAndRetrySet1Async())
.AddPolicyHandler(retryPolicies.WaitAndRetrySet2Async())
.AddPolicyHandler(retryPolicies.WaitAndRetrySet3Async())
.AddPolicyHandler(retryPolicies.BasicCircuitBreakerForAllStatusCodesAsync());
After sometime again I will submit the same requests and I should be able to see the same behaviour. But I am not able to see the desired output.
Could you please describe the observed and the expected behavior?
using Polly.CircuitBreaker; using Polly.Contrib.WaitAndRetry; using Polly.Retry; using Polly; using System.Net;
namespace ClientAPI.TestHarness.Modules { public class WaitAndRetryPolicies {
### Retry1
/// <summary>
/// Retry these codes for 3 times for every second
/// </summary>
private static readonly List<HttpStatusCode> _set1 =
[
HttpStatusCode.Unauthorized //401
];
/// <summary>
/// Policy to retry for 3 times for every second
/// </summary>
public AsyncRetryPolicy<HttpResponseMessage> WaitAndRetrySet1Async()
{
//jitter strategy with ConstantBackoff
var constantBackoff = Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(1), retryCount: 3);
AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
.HandleResult<HttpResponseMessage>(r => _set1.Contains(r.StatusCode))
.Or<BrokenCircuitException>()
.WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetryWaitAndRetrySet1);
return waitAndRetryPolicy;
}
private void OnRetryWaitAndRetrySet1(DelegateResult<HttpResponseMessage> result, TimeSpan timeSpan, int retryCount, Context ctx)
{
Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnRetryWaitAndRetrySet1)} Retry# {retryCount} will be executed after {timeSpan.TotalSeconds} seconds");
}
### Retry2
/// <summary>
/// Retry these codes for 3 times exponentially 1,2,4
/// </summary>
private static readonly List<HttpStatusCode> _set2 =
[
HttpStatusCode.TooManyRequests //429
];
/// <summary>
/// Policy to retry for 3 times for 1,2,4 seconds
/// </summary>
public AsyncRetryPolicy<HttpResponseMessage> WaitAndRetrySet2Async()
{
//jitter strategy with an ExponentialBackoff
var exponentialBackoff = Backoff.ExponentialBackoff(initialDelay: TimeSpan.FromSeconds(1), retryCount: 3);
AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
.HandleResult<HttpResponseMessage>(r => _set2.Contains(r.StatusCode))
.Or<BrokenCircuitException>()
.WaitAndRetryAsync(sleepDurations: exponentialBackoff, onRetry: OnRetryWaitAndRetrySet2);
return waitAndRetryPolicy;
}
private void OnRetryWaitAndRetrySet2(DelegateResult<HttpResponseMessage> result, TimeSpan timeSpan, int retryCount, Context ctx)
{
Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnRetryWaitAndRetrySet2)} Retry# {retryCount} will be executed after {timeSpan.TotalSeconds} seconds");
}
### Retry3
/// <summary>
/// Retry these codes for 3 times every 5 second
/// </summary>
private static readonly List<HttpStatusCode> _set3 =
[
HttpStatusCode.BadGateway, //502
HttpStatusCode.GatewayTimeout //504
];
/// <summary>
/// Policy to retry for 3 times for every 5 second
/// </summary>
public AsyncRetryPolicy<HttpResponseMessage> WaitAndRetrySet3Async()
{
//jitter strategy with ConstantBackoff
var constantBackoff = Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(5), retryCount: 3);
AsyncRetryPolicy<HttpResponseMessage> waitAndRetryPolicy = Policy
.HandleResult<HttpResponseMessage>(r => _set3.Contains(r.StatusCode))
.Or<BrokenCircuitException>()
.WaitAndRetryAsync(sleepDurations: constantBackoff, onRetry: OnRetryWaitAndRetrySet3);
return waitAndRetryPolicy;
}
private void OnRetryWaitAndRetrySet3(DelegateResult<HttpResponseMessage> result, TimeSpan timeSpan, int retryCount, Context ctx)
{
Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnRetryWaitAndRetrySet3)} Retry# {retryCount} will be executed after {timeSpan.TotalSeconds} seconds");
}
}
}
using Polly.CircuitBreaker; using Polly; using System.Net;
namespace ClientAPI.TestHarness.Modules { public class CircuitBreakerPolicy {
### Circuit breaker
/// <summary>
/// Retry these codes for 3 times for every second
/// </summary>
private static readonly List<HttpStatusCode> _set1 =
[
HttpStatusCode.Unauthorized //401
];
/// <summary>
/// Retry these codes for 3 times exponentially 1,2,4
/// </summary>
private static readonly List<HttpStatusCode> _set2 =
[
HttpStatusCode.TooManyRequests //429
];
/// <summary>
/// Retry these codes for 3 times every 5 second
/// </summary>
private static readonly List<HttpStatusCode> _set3 =
[
HttpStatusCode.BadGateway, //502
HttpStatusCode.GatewayTimeout //504
];
/// <summary>
/// This is the circuit breaker policy to work with the above set of all status codes _set1, _set2, _set3.
/// That is a common circuit breaker policy which should work with a set of status codes.
/// </summary>
public AsyncCircuitBreakerPolicy<HttpResponseMessage> CommonCircuitBreakerForAllStatusCodesAsync()
{
AsyncCircuitBreakerPolicy<HttpResponseMessage> basicCircuitBreakerPolicy = Policy
.HandleResult<HttpResponseMessage>(r => _set1.Contains(r.StatusCode) || _set2.Contains(r.StatusCode) || _set3.Contains(r.StatusCode))
.CircuitBreakerAsync(handledEventsAllowedBeforeBreaking: 2,
durationOfBreak: TimeSpan.FromSeconds(10),
onBreak: OnBreak, onReset: OnReset, onHalfOpen: OnHalfOpen);
return basicCircuitBreakerPolicy;
}
private void OnBreak(DelegateResult<HttpResponseMessage> result, TimeSpan timeSpan)
{
Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnBreak)} Circuit transitioned to OPEN state, Duration of break is {timeSpan.TotalSeconds} seconds ");
}
private void OnReset()
{
Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnReset)} Circuit transitioned to CLOSED state");
}
private void OnHalfOpen()
{
Console.WriteLine($"{DateTime.Now.ToString("HH:mm:ss.fff")} {nameof(OnHalfOpen)} Circuit transitioned to Half-Open state");
}
}
}
Hosting starting
info: Microsoft.Hosting.Lifetime[14] Now listening on: http://localhost:5025 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down. info: Microsoft.Hosting.Lifetime[0] Hosting environment: Development dbug: Microsoft.Extensions.Hosting.Internal.Host[2] Hosting started dbug: Microsoft.WebTools.BrowserLink.Net.BrowserLinkMiddleware[1] Response markup is scheduled to include Browser Link script injection. dbug: Microsoft.WebTools.BrowserLink.Net.BrowserLinkMiddleware[2] Response markup was updated to include Browser Link script injection. dbug: ClientAPI.TestHarness.Controllers[0] Begin-Constructor-TestController dbug: ClientAPI.TestHarness.Controllers[0] End-Constructor-TestController dbug: ClientAPI.TestHarness.Controllers[0] [Begin] TestController.HttpListner info: ClientAPI.TestHarness.Controllers[0] 14:48:31.241- Executing the request from user- 502 info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[100] Start processing HTTP request GET http://localhost:6002/502 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/502 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 253.9542ms - 502 14:48:31.571 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds dbug: ClientAPI.TestHarness.Controllers[0] Begin-Constructor-TestController dbug: ClientAPI.TestHarness.Controllers[0] End-Constructor-TestController dbug: ClientAPI.TestHarness.Controllers[0] [Begin] TestController.HttpListner6002 info: ClientAPI.TestHarness.Controllers[0] 14:48:31.953- Executing the request from user- 401 info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[100] Start processing HTTP request GET http://localhost:6002/401 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/401 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 122.093ms - 401 14:48:32.127 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds (mycomment-CORRECT behaviour, since both the user requests are failed, the circuit transitioned to OPEN state because handledEventsAllowedBeforeBreaking=2 for circuit breaker)
14:48:32.128 OnRetryWaitAndRetrySet1 Retry# 1 will be executed after 1 seconds 14:48:33.141 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:48:36.588 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds 14:48:38.155 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds 14:48:41.598 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds 14:48:43.161 OnHalfOpen Circuit transitioned to Half-Open state (mycomment-CORRECT behaviour, circuit was in OPEN state and transitioned to Half-Open state because Duration of break is 10 seconds)
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/401 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 45.0781ms - 401 (mycomment-may be an incorrect behaviour, since the circuit was in OPEN state from 14:48:32 to 14:48:42, transitioned to half open state at 14:48:43 the retry request#1, #2, #3 of the retry policy(WaitAndRetrySet1Async) to the url http://localhost:6002/401 should have been attempted at 14:48:33, 14:48:34, 14:48:35 and expect to fail since the circuit was in OPEN state and no request was allowed to flow. On other hand, the other parallel request http://localhost:6002/502 should have been retried by the policy (WaitAndRetrySet3Async) for 2 times at 14:48:36(retry #1) and 14:48:41(retry #2) and expect these requests to fail since the circuit was in OPEN state and no request was allowed to flow)
14:48:43.216 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds (mycomment-CORRECT behaviour, circuit transitioned from half-open to OPEN state since the last request was failed)
14:48:43.216 OnRetryWaitAndRetrySet1 Retry# 2 will be executed after 1 seconds 14:48:44.220 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:48:46.613 OnRetryWaitAndRetrySet2 Retry# 1 will be executed after 1 seconds 14:48:47.620 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:48:49.227 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds 14:48:52.636 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds
(mycomment-may be an incorrect behaviour since this log should not be printed(OnRetryWaitAndRetrySet2) as I have not recieved any response which belongs to OnRetryWaitAndRetrySet2 method)
(mycomment-may be an incorrect behaviour, since the circuit was in OPEN state from 14:48:43 to 14:48:53, transitioned to half open state at 14:48:54 the retry request #3 & #4 should have been retried at 14:48:46 and at 14:48:51 or 52 by the policy (WaitAndRetrySet3Async) to the url http://localhost:6002/502 and is exptected to fail since circuit was in OPEN state and no request was allowed to flow)
14:48:54.227 OnHalfOpen Circuit transitioned to Half-Open state (mycomment-CORRECT behaviour, circuit was in OPEN state and transitioned to Half-Open state because Duration of break is 10 seconds)
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/401 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 35.1538ms - 401
14:48:54.268 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds (mycomment-CORRECT behaviour, circuit transitioned from half-open to OPEN state since the last request was failed)
14:48:54.269 OnRetryWaitAndRetrySet1 Retry# 3 will be executed after 1 seconds 14:48:55.277 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:48:57.647 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds 14:49:00.291 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds 14:49:02.660 OnRetryWaitAndRetrySet2 Retry# 2 will be executed after 2 seconds 14:49:04.668 OnHalfOpen Circuit transitioned to Half-Open state (mycomment-CORRECT behaviour, circuit was in OPEN state and transitioned to Half-Open state because Duration of break is 10 seconds)
(mycomment-may be an incorrect behaviour, the retry request #5 should have been retried at 14:48:56 and is exptected to fail since circuit was in OPEN state and no request was allowed to flow)
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/502 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 51.3886ms - 502 14:49:04.729 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds (mycomment-CORRECT behaviour, circuit transitioned from half-open to OPEN state since the last request was failed)
14:49:04.729 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:49:05.312 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds 14:49:09.745 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds 14:49:10.327 OnRetryWaitAndRetrySet2 Retry# 1 will be executed after 1 seconds 14:49:11.341 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:49:14.754 OnHalfOpen Circuit transitioned to Half-Open state (mycomment-CORRECT behaviour, circuit was in OPEN state and transitioned to Half-Open state because Duration of break is 10 seconds)
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/502 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 33.7128ms - 502 14:49:14.795 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds 14:49:14.795 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds 14:49:16.343 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds 14:49:19.802 OnRetryWaitAndRetrySet2 Retry# 3 will be executed after 4 seconds 14:49:21.354 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds 14:49:23.811 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:49:26.360 OnHalfOpen Circuit transitioned to Half-Open state (mycomment-CORRECT behaviour, circuit was in OPEN state and transitioned to Half-Open state because Duration of break is 10 seconds)
info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/401 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 50.1859ms - 401 14:49:26.416 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[101] End processing HTTP request after 54460.8472ms - 401 info: ClientAPI.TestHarness.Controllers[0] 14:49:26.423 [End] TestController.HttpListner6002- The service responded with 401-Unauthorized status code for the request URI http://localhost:6002/401. All the retries are completed. dbug: ClientAPI.TestHarness.Controllers[0] 14:49:26- [End] TestController.HttpListner6002 14:49:28.829 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds 14:49:33.853 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds 14:49:38.858 OnHalfOpen Circuit transitioned to Half-Open state info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/502 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 37.823ms - 502 14:49:38.904 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[101] End processing HTTP request after 67644.8797ms - 502 info: ClientAPI.TestHarness.Controllers[0] 14:49:38.908 [End] TestController.HttpListner- The service responded with 502-Bad Gateway status code for the request URI http://localhost:6002/502. All the retries are completed. dbug: ClientAPI.TestHarness.Controllers[0] 14:49:38- [End] TestController.HttpListner
======================2nd time submitted the same requests again============================
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
Ending HttpMessageHandler cleanup cycle after 0.005ms - processed: 0 items - remaining: 1 items
dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100] Starting HttpMessageHandler cleanup cycle with 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101] Ending HttpMessageHandler cleanup cycle after 0.0094ms - processed: 0 items - remaining: 1 items dbug: ClientAPI.TestHarness.Controllers[0] Begin-Constructor-TestController dbug: ClientAPI.TestHarness.Controllers[0] End-Constructor-TestController dbug: ClientAPI.TestHarness.Controllers[0] [Begin] TestController.HttpListner info: ClientAPI.TestHarness.Controllers[0] 14:54:06.173- Executing the request from user- 502 info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[100] Start processing HTTP request GET http://localhost:6002/502 14:54:06.187 OnHalfOpen Circuit transitioned to Half-Open state info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/502 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 138.088ms - 502 14:54:06.340 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds (mycomment-CORRECT behaviour, circuit transitioned from half-open to OPEN state since the last request was failed)
14:54:06.341 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds dbug: ClientAPI.TestHarness.Controllers[0] Begin-Constructor-TestController dbug: ClientAPI.TestHarness.Controllers[0] End-Constructor-TestController dbug: ClientAPI.TestHarness.Controllers[0] [Begin] TestController.HttpListner6002 info: ClientAPI.TestHarness.Controllers[0] 14:54:06.913- Executing the request from user- 401 info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[100] Start processing HTTP request GET http://localhost:6002/401 14:54:06.920 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:54:11.332 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100] Starting HttpMessageHandler cleanup cycle with 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101] Ending HttpMessageHandler cleanup cycle after 0.0063ms - processed: 0 items - remaining: 1 items 14:54:11.920 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds 14:54:16.340 OnHalfOpen Circuit transitioned to Half-Open state info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/502 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 44.898ms - 502 14:54:16.395 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds 14:54:16.395 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds 14:54:16.920 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds 14:54:21.408 OnRetryWaitAndRetrySet2 Retry# 1 will be executed after 1 seconds dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100] Starting HttpMessageHandler cleanup cycle with 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101] Ending HttpMessageHandler cleanup cycle after 0.005ms - processed: 0 items - remaining: 1 items 14:54:21.925 OnRetryWaitAndRetrySet2 Retry# 1 will be executed after 1 seconds 14:54:22.422 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:54:22.928 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:54:27.427 OnHalfOpen Circuit transitioned to Half-Open state info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/502 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 50.187ms - 502 14:54:27.486 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds 14:54:27.486 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds 14:54:27.936 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100] Starting HttpMessageHandler cleanup cycle with 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101] Ending HttpMessageHandler cleanup cycle after 0.0057ms - processed: 0 items - remaining: 1 items 14:54:32.499 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds 14:54:32.935 OnRetryWaitAndRetrySet3 Retry# 3 will be executed after 5 seconds 14:54:37.499 OnHalfOpen Circuit transitioned to Half-Open state info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/502 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 36.1713ms - 502 14:54:37.541 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[101] End processing HTTP request after 31387.3779ms - 502 info: ClientAPI.TestHarness.Controllers[0] 14:54:37.544 [End] TestController.HttpListner- The service responded with 502-Bad Gateway status code for the request URI http://localhost:6002/502. All the retries are completed. dbug: ClientAPI.TestHarness.Controllers[0] 14:54:37- [End] TestController.HttpListner 14:54:37.942 OnRetryWaitAndRetrySet2 Retry# 2 will be executed after 2 seconds 14:54:39.957 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100] Starting HttpMessageHandler cleanup cycle with 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101] Ending HttpMessageHandler cleanup cycle after 0.0507ms - processed: 0 items - remaining: 1 items 14:54:44.970 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds 14:54:49.981 OnHalfOpen Circuit transitioned to Half-Open state info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/401 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 193.4952ms - 401 14:54:50.329 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds 14:54:50.329 OnRetryWaitAndRetrySet1 Retry# 1 will be executed after 1 seconds 14:54:51.329 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100] Starting HttpMessageHandler cleanup cycle with 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101] Ending HttpMessageHandler cleanup cycle after 0.0041ms - processed: 0 items - remaining: 1 items 14:54:56.343 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds 14:55:01.338 OnHalfOpen Circuit transitioned to Half-Open state info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/401 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 40.0117ms - 401 14:55:01.384 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds 14:55:01.385 OnRetryWaitAndRetrySet1 Retry# 2 will be executed after 1 seconds dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100] Starting HttpMessageHandler cleanup cycle with 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101] Ending HttpMessageHandler cleanup cycle after 0.0043ms - processed: 0 items - remaining: 1 items 14:55:02.400 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:55:07.406 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100] Starting HttpMessageHandler cleanup cycle with 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101] Ending HttpMessageHandler cleanup cycle after 0.0053ms - processed: 0 items - remaining: 1 items 14:55:12.408 OnHalfOpen Circuit transitioned to Half-Open state info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/401 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 32.8548ms - 401 14:55:12.445 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds 14:55:12.446 OnRetryWaitAndRetrySet1 Retry# 3 will be executed after 1 seconds 14:55:13.454 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 5 seconds 14:55:18.452 OnRetryWaitAndRetrySet3 Retry# 2 will be executed after 5 seconds dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100] Starting HttpMessageHandler cleanup cycle with 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101] Ending HttpMessageHandler cleanup cycle after 0.0255ms - processed: 0 items - remaining: 1 items 14:55:23.461 OnHalfOpen Circuit transitioned to Half-Open state info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[100] Sending HTTP request GET http://localhost:6002/401 info: System.Net.Http.HttpClient.HttpListener6002.ClientHandler[101] Received HTTP response headers after 59.3668ms - 401 14:55:23.528 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds info: System.Net.Http.HttpClient.HttpListener6002.LogicalHandler[101] End processing HTTP request after 76684.2992ms - 401 info: ClientAPI.TestHarness.Controllers[0] 14:55:23.564 [End] TestController.HttpListner6002- The service responded with 401-Unauthorized status code for the request URI http://localhost:6002/401. All the retries are completed. dbug: ClientAPI.TestHarness.Controllers[0] 14:55:23- [End] TestController.HttpListner6002 dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100] Starting HttpMessageHandler cleanup cycle with 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101] Ending HttpMessageHandler cleanup cycle after 0.0042ms - processed: 0 items - remaining: 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100] Starting HttpMessageHandler cleanup cycle with 1 items dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101] Ending HttpMessageHandler cleanup cycle after 0.0052ms - processed: 0 items - remaining: 1 items
I have written some comments(mycomment-) in the log. Can you please review it.
This is how I applied the policies at startup.cs
WaitAndRetryPolicies waitAndRetryPolicies = new WaitAndRetryPolicies(); CircuitBreakerPolicy circuitBreakerPolicy = new CircuitBreakerPolicy();
var retry1 = waitAndRetryPolicies.WaitAndRetrySet1Async(); var retry2 = waitAndRetryPolicies.WaitAndRetrySet2Async(); var retry3 = waitAndRetryPolicies.WaitAndRetrySet3Async();
var allRetries = Policy.WrapAsync(retry1, retry2, retry3); var commonCircuitBreaker = allRetries.WrapAsync(circuitBreakerPolicy.CommonCircuitBreakerForAllStatusCodesAsync());
//Creating the named client instance for HttpListener6002 builder.Services.AddHttpClient("HttpListener6002", client => { client.BaseAddress = new Uri(http://localhost:6002/); //BaseAddress for named client instance HttpListener6002 }) .AddPolicyHandler(commonCircuitBreaker);
This is my problem statement
Define policies for the following use cases 1)Status code 401 retry for 3 times for every second 2)Status code 423 retry for 3 times exponentially 1,2,4 seconds 3)Status codes 502/504 retry for 3 times for every 5 second
I would like to define the retry policies for the above 3 use cases and I would like to define a common circuit breaker policy which should work for all these 3 use cases for multiple simultaneous requests for a single named http client instance.
Can you please correct me if anything is wrong in my code or please point me to right solution to achieve the same. Thank you in advance.
@Mahantesh-DotNet No, your implementation does not work correctly. Let me demonstrate it.
Here is a simplified version of your code:
public static async Task Main()
{
var strategy = Policy.WrapAsync<HttpResponseMessage>(RetryScenario1(), RetryScenario2(), RetryScenario3(), CircuitBreaker());
await strategy.ExecuteAsync(() => Task.FromResult(new HttpResponseMessage(HttpStatusCode.Unauthorized)));
}
static IAsyncPolicy<HttpResponseMessage> RetryScenario1()
=> Policy<HttpResponseMessage>
.HandleResult(r => r.StatusCode == HttpStatusCode.Unauthorized)
.Or<BrokenCircuitException>()
.WaitAndRetryAsync(sleepDurations: Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(1), retryCount: 3),
onRetry: (ex, ts) => Console.WriteLine("OnRetry for 401"));
static IAsyncPolicy<HttpResponseMessage> RetryScenario2()
=> Policy<HttpResponseMessage>
.HandleResult(r => r.StatusCode == HttpStatusCode.TooManyRequests)
.Or<BrokenCircuitException>()
.WaitAndRetryAsync(sleepDurations: Backoff.ExponentialBackoff(initialDelay: TimeSpan.FromSeconds(1), retryCount: 3),
onRetry: (ex, ts) => Console.WriteLine("OnRetry for 429"));
static IAsyncPolicy<HttpResponseMessage> RetryScenario3()
=> Policy<HttpResponseMessage>
.HandleResult(r => r.StatusCode == HttpStatusCode.BadGateway || r.StatusCode == HttpStatusCode.GatewayTimeout)
.Or<BrokenCircuitException>()
.WaitAndRetryAsync(sleepDurations: Backoff.ConstantBackoff(delay: TimeSpan.FromSeconds(5), retryCount: 3),
onRetry: (ex, ts) => Console.WriteLine("OnRetry for 502, 504"));
static List<HttpStatusCode> retryable = [ HttpStatusCode.Unauthorized, HttpStatusCode.TooManyRequests, HttpStatusCode.BadGateway, HttpStatusCode.GatewayTimeout ];
static IAsyncPolicy<HttpResponseMessage> CircuitBreaker()
=> Policy<HttpResponseMessage>
.HandleResult(r => retryable.Contains(r.StatusCode))
.CircuitBreakerAsync(handledEventsAllowedBeforeBreaking: 2,
durationOfBreak: TimeSpan.FromSeconds(10),
onBreak: (r, ts) => Console.WriteLine("OnBreak"),
onReset: () => Console.WriteLine("OnReset"),
onHalfOpen: () => Console.WriteLine("OnHalfOpen"));
If I run this code inside dotnet fiddle then I receive the following output:
OnRetry for 401
OnBreak
OnRetry for 401
OnRetry for 502, 504
OnRetry for 502, 504
So, why do we have OnRetry for 502, 504 if we always return 401?
Well the reason is that your retry policies triggers for BrokenCircuitException as well. So, when the CB is open then it shortcuts the execution by throwing a BrokenCircuitException. The next policy in the policy chain is the RetryScenario3. That handles this exception that's why we see OnRetry for 502, 504.
One way to solve this problem:
- Remove the
Or<BrokenCircuitException>()clauses from your retry policies - Define a 4th policy which handles only the
BrokenCircuitException
By doing that all four retry policies become mutually exclusive. In other words, their registration order inside the policy chain does not matter because they trigger under different circumstances.
Thank you so much for your inputs. I commented out //.Or<BrokenCircuitException>() from all my retry policies and defined a new policy to handle the BrokenCircuitException as below
public AsyncRetryPolicy NoWaitAndRetryAsync() { AsyncRetryPolicy noWaitAndRetryPolicy = Policy .Handle<BrokenCircuitException>() .RetryAsync(0); return noWaitAndRetryPolicy; }
var allRetries = Policy.WrapAsync<HttpResponseMessage>(RetryScenario1(), RetryScenario2(), RetryScenario3(), waitAndRetryPolicies.NoWaitAndRetryAsync().AsAsyncPolicy<HttpResponseMessage>(), CircuitBreaker());
I see an exception during the first attempt itself. Not sure what's wrong here. Please help.
14:50:56.239 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
14:50:56.240 OnBreak Circuit transitioned to OPEN state, Duration of break is 10 seconds
14:50:56.242 OnRetryWaitAndRetrySet3 Retry# 1 will be executed after 1 seconds
14:50:56.242 OnRetryWaitAndRetrySet1 Retry# 1 will be executed after 1 seconds
fail: ClientAPI.TestHarness.Controllers[0]
[Exception] The circuit is now open and is not allowing calls.
fail: ClientAPI.TestHarness.Controllers[0]
[Exception] The circuit is now open and is not allowing calls.
fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
An unhandled exception has occurred while executing the request.
Polly.CircuitBreaker.BrokenCircuitException1[System.Net.Http.HttpResponseMessage]: The circuit is now open and is not allowing calls. at Polly.CircuitBreaker.CircuitStateController1.OnActionPreExecute()
Whenever the CB is open then you don't have to retry immediately because it will fail with a BrokenCircuitException.
I would recommend the following approach instead:
static IAsyncPolicy<HttpResponseMessage> RetryBrokenCircuit()
=> Policy<HttpResponseMessage>
.Handle<BrokenCircuitException>()
.WaitAndRetryForeverAsync(sleepDurationProvider: _ => TimeSpan.FromSeconds(5),
onRetry: (ex, ts) => Console.WriteLine("OnRetry for BCE"));
- I would wait half as much time as the CB's break duration. (in the example: 10 sec / 2 = 5 sec)
- I would not limit the number of retries for the BCE (WaitAndRetryForever)
- Depending on your requirements it may or may not make sense
If I let the application running until it exhausts all the retry attempts for the RetryScenario1 then the output:
OnRetry for 401
OnBreak
OnRetry for 401
OnRetry for BCE
OnRetry for BCE
OnRetry for BCE
OnHalfOpen
OnBreak
OnRetry for 401
OnRetry for BCE
OnRetry for BCE
OnRetry for BCE
OnHalfOpen
OnBreak
Hi @peter-csala thank you for you inputs. I am trying your suggestions. Currently I am testing all these policies with 2 simulaneous http requests for the same http client. I might face challenges, I will let you know after I succeed with this test. Once again thank you for your support.
This issue is stale because it has been open for 60 days with no activity. It will be automatically closed in 14 days if no further updates are made.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue was closed because it has been inactive for 14 days since being marked as stale.