gh-gei icon indicating copy to clipboard operation
gh-gei copied to clipboard

AdoClient should retry all failures on GetWithPagingAsync

Open dylan-smith opened this issue 3 years ago • 5 comments

Currently AdoClient retries all failures for GetAsync() but GetWithPagingAsync() only retries on 503's. Ideally we would have GetWithPagingAsync() call GetAsync() so that the behavior matches.

This would likely have fixed this integration test failure: https://github.com/github/gh-gei/actions/runs/3381867801/jobs/5616208755#step:18:48

dylan-smith avatar Nov 03 '22 01:11 dylan-smith

Another broken build affected: https://github.com/github/gh-gei/actions/runs/3382698823/jobs/5617886608#step:14:547

dylan-smith avatar Nov 03 '22 03:11 dylan-smith

Another one: https://github.com/github/gh-gei/actions/runs/3904289936/jobs/6672691652#step:19:4308

dylan-smith avatar Jan 12 '23 20:01 dylan-smith

another example: https://github.com/github/gh-gei/actions/runs/4450558388/jobs/7816162392#step:19:1844

dylan-smith avatar Mar 17 '23 21:03 dylan-smith

another one with a SocketException/IOException: https://github.com/github/gh-gei/actions/runs/4670751380/jobs/8271717864#step:12:348

[xUnit.net 00:07:51.35]     OctoshiftCLI.IntegrationTests.AdoCsvToGithub.With_Inventory_Report_Csv [FAIL]
  Failed OctoshiftCLI.IntegrationTests.AdoCsvToGithub.With_Inventory_Report_Csv [7 m 49 s]
  Error Message:
   System.Net.Http.HttpRequestException : An error occurred while sending the request.
---- System.IO.IOException : Unable to read data from the transport connection: Connection reset by peer.
-------- System.Net.Sockets.SocketException : Connection reset by peer
  Stack Trace:
     at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at OctoshiftCLI.Services.AdoClient.<>c__DisplayClass13_0.<<GetWithPagingAsync>b__0>d.MoveNext() in /Users/runner/work/gh-gei/gh-gei/src/Octoshift/Services/AdoClient.cs:line 121
--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at OctoshiftCLI.RetryPolicy.HttpRetry[T](Func`1 func, Func`2 filter) in /Users/runner/work/gh-gei/gh-gei/src/Octoshift/RetryPolicy.cs:line 29
   at OctoshiftCLI.Services.AdoClient.GetWithPagingAsync(String url, String continuationToken) in /Users/runner/work/gh-gei/gh-gei/src/Octoshift/Services/AdoClient.cs:line 119
   at OctoshiftCLI.Services.AdoClient.GetWithPagingAsync(String url) in /Users/runner/work/gh-gei/gh-gei/src/Octoshift/Services/AdoClient.cs:line 93
   at OctoshiftCLI.Services.AdoApi.GetIdentityDescriptor(String org, String teamProjectId, String groupName) in /Users/runner/work/gh-gei/gh-gei/src/Octoshift/Services/AdoApi.cs:line 652
   at OctoshiftCLI.IntegrationTests.TestHelper.AssertAdoRepoLocked(String adoOrg, String teamProject, String repo) in /Users/runner/work/gh-gei/gh-gei/src/OctoshiftCLI.IntegrationTests/TestHelper.cs:line 629
   at OctoshiftCLI.IntegrationTests.AdoCsvToGithub.With_Inventory_Report_Csv() in /Users/runner/work/gh-gei/gh-gei/src/OctoshiftCLI.IntegrationTests/AdoCsvToGithub.cs:line 56

dylan-smith avatar Apr 11 '23 19:04 dylan-smith

Another failure this would have prevented: https://github.com/github/gh-gei/actions/runs/5398224828/jobs/9803881769#step:16:151

dylan-smith avatar Jun 28 '23 15:06 dylan-smith