Http Stress Status Report

What we've run so far:

OS	HTTP 1.1	HTTP 2.0	Notes
Windows	4+ h (Mana) -> 2 errors	12 h (JJ) -> 11 errors
	6 h (Miha) -> 8 errors	6 h (Miha) -> 7 errors
	10 h (Miha) -> 53 errors	7 h (Miha) -> 139 errors	7 h = 1 + 6 h
Linux	17 h (JJ) -> 0 errors	12 h (JJ) -> 608 errors	we should rerun as this may be environmental problem
	12 h (Furtik) -> 0 errors	12 h (Furtik) -> 104 errors	ran on released runtime (not on master)

HTTP 2.0 Error Statistics

Error Type	Linux	Windows
Success	135,727,755	105,273,552
Errors	712	157
System.Threading.Tasks.TaskCanceledException: The operation was canceled.	37	8
System.Threading.Tasks.TaskCanceledException: A task was canceled.	13	4
System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.	464	18
System.Net.Sockets.SocketException (32): Broken pipe	198	0
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.	0	118
System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.	0	9

HTTP 1.1 Error Statistics

Error Type	Linux	Windows
Success	171,673,491	128,658,822
Errors	0	61
System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.	0	61

What we need to run:

More HTTP 1.1 Linux runs to confirm that we're clear. (easy, hi pri)
More HTTP 2.0 Linux runs to confirm that we have all error types captured. (easy, hi pri)
HTTP 2.0 tests against HTTPSys to eliminate/confirm server as the culprit. (mid, mid pri)
Run the matrix against 3.1 and compare. (hard, mid pri)

Existing issues, root caused:

HTTP 1.1 connect cancellation deadlock: https://github.com/dotnet/runtime/issues/42198
HTTP 2.0 sending RESET before END_STREAM: https://github.com/dotnet/runtime/issues/42200
HTTP 2.0 throwing TaskCancelledException as a reaction on GO_AWAY: https://github.com/dotnet/runtime/issues/42472
NEW HTTP 2.0 infinite loop when connecting H2C to HTTP 1.0 localhost server: https://github.com/dotnet/runtime/issues/42259

Discovered exceptions, not-investigated:

HTTP 2.0 System.Net.Sockets.SocketException (32): Broken pipe. (Linux only)
HTTP 2.0 System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.
~~HTTP 2.0 System.Threading.Tasks.TaskCanceledException: The operation was canceled.~~ https://github.com/dotnet/runtime/issues/42472
HTTP 2.0 System.Threading.Tasks.TaskCanceledException: A task was canceled. --> might be covered by https://github.com/dotnet/runtime/issues/42472
HTTP 2.0 System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host. (Windows only)
HTTP 2.0 System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine. (Windows only)
NEW HTTP 1.1 System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (Windows only)

The discovered exceptions confirm what we've collected so far from the pipelines: https://github.com/dotnet/runtime/issues/40388.

Distributable tasks by priority:

More HTTP 1.1 Linux runs: http11run
More HTTP 2.0 Linux runs: http20run
Investigate Windows HTTP 1.1 errors: winErr3
Investigate Windows HTTP 2.0 errors: winErr1, winErr2
Provide fix for #42200
Provide fix for #42198
Help with HTTPSys client connection errors: httpSys: put on back-burner
Run the tests against .NET Core 3.1: net31: put on back-burner

Tips and Tricks for investigations:

Clear up docker containers and images after a product code change (docker container prune && docker image prune -a)
Once container is built, switch -b might be omitted for subsequent re-runs (skips the runtime build)
Don't use containers for investigations, they're slow and rebuild takes too long
- To run the app against locally built runtime, swap artifacts/bin/testhost/net5.0-Linux-Debug-x64/shared/Microsoft.NETCore.App/6.0.0/ with the globally installed runtime (/usr/share/dotnet/shared/Microsoft.NETCore.App/your-latest-5.0-version)
  - Make a backup of the global runtime!
  - Using testhost's corerun didn't work for me since the app depends on ASP .NET Core SDK
  - If you change the product code, rebuild just System.Net.Http and copy System.Net.Http.dll to the global runtime again
- Build System.Net.Http/tests/StressTests/HttpStress
- Open 2 terminals and run:
  - server: dotnet run -runMode server -aspnetlog
    - -aspnetlog: console logging of server errors
    - -serverUri https://localhost:5002: bind to a different port (when running multiple tests in parallel)
  - client: dotnet run -runMode client
    - -serverUri https://localhost:5002: connect to a different port (when running multiple tests in parallel)
    - -ops 1 2 3: run only operation 1, 2 and 3 (GET, PUT Slow, etc...)
    - -trace: saves internal client/server traces in a log file, very verbose, useable only for very short runs
  - more options in source code: https://github.com/dotnet/runtime/blob/master/src/libraries/System.Net.Http/tests/StressTests/HttpStress/Program.cs#L37-L62

If you have any improvements to the stress app or the containers, please create a PR and don't keep it just for yourself. If you have more tips and tricks for running the tests, please share them.

Sep 14 '20 18:09 ManickaP

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Sep 14 '20 18:09 msftbot[bot]

The description of #49999 has some useful links to stack traces according how failures occur today. The "new" HTTP 1.1 connection timeout seems to be a dominant issue on CI, failing on almost every Windows run.

Mar 22 '21 16:03 antonfirsov

In the last 20 CI runs, there were 3 occurrences of the "System.Net.Sockets.SocketException (10054)" Windows + HTTP 2.0 case. I won't be surprised, if it's also a firewall issue being eventually solved by #52381. If not we should look into it.

May 10 '21 11:05 antonfirsov

I browsed through the failures of the last 25 days in the CI, and it seems that we are facing the same errors as las September. The only thing worth mentioning is that I've noticed a TaskCanceledException failure with a stack trace that comes from PUT Slow, instead of POST ExpectContinue I can't decide if it's covered by #42472 or not, we may want to check it:

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
-> System.TimeoutException: A task was canceled.
-> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.Http2Connection.SendStreamDataAsync(Int32 streamId, ReadOnlyMemory`1 buffer, Boolean finalFlush, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1482
at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 1275
at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 623
at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 291
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1941
at System.Net.Http.HttpConnectionPool.TrySendUsingHttp2Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 833
at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 897
at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 910
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 920
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 548

Full CI run

Jun 28 '21 10:06 antonfirsov

@antonfirsov can you please add note about your analysis from last week?

Jul 29 '21 08:07 karelz

Report 7/29

with manual observations after a brief look through the CI output of between 7/13-7/29, excluding the HTTP3 failures:

There is a new failure type now tracked under #56159
Other failures I've seen are known and described in the opening post
There are 2 cases where the Job run is unfinished. Likely an AZP anomaly, but we should keep monitoring if it happens again

Going back a bit more I see signs of #55261 (see here), but it doesn't seem to happen since 7/8 (EDIT: Because the test was disabled on 7/8).

Jul 29 '21 14:07 antonfirsov

Report 8/5

brief look through the CI output between 7/29-8/5:

No new failure types in the test runs
#56775 broke the last build, which is being fixed in #56912

Aug 05 '21 15:08 antonfirsov

Report 8/13

No new failure types in HTTP 1.1 & 2.0 in the last 1 week.

Aug 13 '21 09:08 antonfirsov

Report 8/24

No new failure types in HTTP 1.1 & 2.0 since last report
There is one ~occurrence~ occurence of an unexpected ObjectDisposedException that happened after merging #57287. I'm reluctant to reopen #56159 right away, will do local runs and keep watching the CI, and open an issue if it happens again.

Aug 24 '21 15:08 antonfirsov

The occurrence is on 13.8., isn't that before the fix got merged?

Aug 24 '21 15:08 ManickaP

Sorry, linked wrong build, this one happened on the 18th: https://dev.azure.com/dnceng/public/_build/results?buildId=1301334&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=45b31e99-b18c-506f-46cf-7d3a4d5140a9

Aug 24 '21 16:08 antonfirsov

System.Net.Sockets.SocketException (32): Broken pipe. (Linux only)

I am getting this issue when I stress test my application. My application is running in docker linux container. What is the cause of this issue? Is there a solution/workaround for this?

Here is the exception chain

HttpRequestException: Error while copying content to a stream. SocketException: Unable to read data from the transport connection: Broken pipe. IOException: Broken pipe

Aug 25 '21 10:08 Vijay-Nirmal

@Vijay-Nirmal please file a separate issue with additional information. This issue is tracking general stress test runs, which may or may not be related to what you see. It is fine to link this issue. Thanks!

Aug 25 '21 12:08 karelz

Report 2021/8/31:

No new failure types, and no HTTP-level failures in 1.1/2.0 during the last week, although there are two occurrences of the 10060 (WSAETIMEDOUT) socket error despite firewall being disabled.

Aug 31 '21 14:08 antonfirsov

Report 2021/9/14:

Checked both main and release/6.0.

No new failure types
All HTTP errors are gone
About 10% of the HTTP 1.1 Windows runs fails with SocketException (10060), which makes it clear that disabling the firewall is insufficient to fix that problem. Opened #59086 to track this.

Sep 14 '21 13:09 antonfirsov

Report 2022/8/1:

During the last week, all HTTP 1.1 and 2.0 Linux runs were successful on the CI
The Windows pipeline suffers from build failures, recently from #73154
HTTP/3 reports vary heavily (some runs green, others with 157 or even 8000 fauilures. I'm not sure if all those failures are tracked or if we aim for green runs for 7.0 at all. /cc @ManickaP @CarnaViire

Aug 01 '22 14:08 antonfirsov

SslStress 2022/8/3

Since #72940 there were 6 SslStress runs on Linux, all green. The Windows runs suffer from #73154.

Aug 03 '22 19:08 antonfirsov

Report 2022/8/9:

As of 2022/8/5, #73154 has been resolved. The Windows pipeline is building again. One nightly run failed to build on Windows (logs), though it looks like a temporary issue with fetching packages from NuGet that has since been resolved
All HTTP/1.1 and HTTP/2 runs were successful on both Windows and Linux

SslStress 2022/8/9

All 5 Linux runs are green
3 out of the 5 Windows runs had failures (7 request failures in total) with a similar SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. error (logs, logs, logs)
All runs print a ton of noise onto the console from unhandled server exceptions (example) - do we care?

Aug 09 '22 21:08 MihaZupan

Moving to 8.0 as it is just tracking reports ...

Aug 12 '22 16:08 karelz

Stress tests are broken after 7.0 snap: example log

client_1  | Framework: 'Microsoft.NETCore.App', version '7.0.0' (x64)
client_1  | .NET location: /live-runtime-artifacts/testhost/net7.0-Linux-Release-x64/
client_1  | 
client_1  | The following frameworks were found:
client_1  |   8.0.0 at [/live-runtime-artifacts/testhost/net7.0-Linux-Release-x64/shared/Microsoft.NETCore.App]

Also, we need to set up regular runs for release/7.0, I guess that doesn't update automatically.

Aug 18 '22 17:08 CarnaViire

Report 2022/9/5:

Unfortunately, I noticed late that #74422 was not sufficient to enable 7.0 runs. Because of that we lost 2 weeks of stress run info :( There is a backport PR #75082 that has to be merged to have 7.0 runs working (hopefully).

There are a few data points of from manual runs against 7.0 and the current main (which is still mostly identical): 1, 2, 3, 4, 5

What we see here:

GET Aborted HTTP/3 failures as expected
An HTTP/3 crash with a dump: DumpsLinux.zip
SslStress failure on Windows with SocketException (995) like in https://github.com/dotnet/runtime/issues/42211#issuecomment-1209893611

The fix for runs against main is up: #75026

Sep 05 '22 16:09 antonfirsov

@CarnaViire is the crash I noticed #72696 or a new issue?

https://dev.azure.com/dnceng-public/public/_build/results?buildId=3446&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd

Sep 06 '22 16:09 antonfirsov

Yes, that's https://github.com/dotnet/runtime/issues/72696. The fix is not merged yet, so it is bound to appear from time to time.

Sep 06 '22 16:09 CarnaViire

Report 2022/9/14:

release/7.0
- All runs succeeded with no failures across all HTTP versions.
main
- One run failed to build on Windows (looks like a transient NuGet issue again).
- There was a single request failure on one HTTP/3 run: content mismatch. It appears the content was truncated somehow.

SslStress:

release/7.0
- No failures on Linux.
- Windows runs consistently report a handful of SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. errors like in previous runs. There were no other failures.
main
- On Linux, 4 request on a single run failed with SocketException (111): Connection refused. No SSL failures.
- Same as release/7.0 on Windows - a handful of SocketException (995) failures.

Sep 14 '22 21:09 MihaZupan

Report 2022/9/26

HttpStress

All Windows runs are broken since 09-21 on both main and release/7.0, opened #76045.

release/7.0
- One of the HTTP/3 runs failed to bootstrap reporting lots of timeouts and a process crash (?). This looks like a transient issue, but if it happens again we should investigate.
main
- The content mismatch issue noticed in the previous report occured again. We should investigate: #76183

SslStress

Most runs are affected by #76044, on both branches.

Sep 26 '22 21:09 antonfirsov

Report 2022/10/17

HttpStress

release/7.0
- Forgot to backport #76307, therefore there was no data on Windows runs in this period. This should be fixed with #77125.
- There were several HTTP/3 failures /cc @CarnaViire:
  - Two occurrences of #77126
  - 1 x OperationCanceledException in POST Duplex
  - 1 x QuicException with Connection aborted by peer (261): https://dev.azure.com/dnceng-public/public/_build/results?buildId=39894&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd
main
- One occurrence of #77126, otherwise everything looks green!

SslStress

Most runs are affected by #76044, on both branches.
There was an occurrence of a different socket failure (10054 - WSAECONNRESET)

Oct 17 '22 20:10 antonfirsov

Report 2022/11/22

Analyzing stress runs in the last ~30 days.

HttpStress

main
- Linux: 28/30 runs were successful. Failures are:
  - #76183
  - A new failure The HTTP/3 server closed the connection. HTTP/3 error code 'H3_FRAME_UNEXPECTED', occured only once

Stack Trace for the H3_FRAME_UNEXPECTED failure

client_1  |  ---> System.Net.Http.HttpProtocolException: The HTTP/3 server closed the connection. HTTP/3 error code 'H3_FRAME_UNEXPECTED' (0x105).
client_1  |    --- End of inner exception stack trace ---
client_1  |    at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 273
client_1  |    at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 320
client_1  |    at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 237
client_1  |    at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 249
client_1  |    at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 990
client_1  |    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1034
client_1  |    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
client_1  |    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 532
client_1  |    at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /app/ClientOperations.cs:line 95
client_1  |    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_13>d.MoveNext() in /app/ClientOperations.cs:line 453
client_1  | --- End of stack trace from previous location ---
client_1  |    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /app/StressClient.cs:line 204

Windows: 28/28 successful runs
release/7.0
- Linux: 14/16 runs were succesful. Failures are: #76183, #77126
- Windows: 13/13 successful runs

SslStress

main
- Linux: 30/30 successful runs
- Windows 17/31 runs failed with #76044
release/7.0
- Linux: 16/16 succesful runs
- Windows: 10/16 runs failed with #76044

Nov 22 '22 19:11 antonfirsov

Report 2023/02/06

Looking at main only.

We had major build failures which got fixed approximately 2 weeks ago in #80713, looking at the runs at this recent period now.

HttpStress

All runs were green on main until another build break landed recently on Linux: #81706

SslStress

Usual occurances of #76044 in 50% of the runs, no new failures.

Feb 06 '23 21:02 antonfirsov

Report 2023/03/28

Main only. Looking at last 13 runs since the fix of #82528.

HttpStress

Linux

2 x #76183
2 x #77126

Windows

1 x QuicException with Stream aborted by peer (267):

System.Net.Http.HttpRequestException: Error while copying content to a stream.
---> System.Net.Quic.QuicException: Stream aborted by peer (267).
  at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
  at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 221
  at System.Net.Quic.QuicStream.HandleEventPeerReceiveAborted(_PEER_RECEIVE_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 590
  at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 669
  at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 695
2023-03-20T13:37:38.9592355Z httpstress-client-1  | --- End of stack trace from previous location ---
  at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 235
  at System.Net.Http.Http3RequestStream.WriteRequestContentAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 479
  at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 615
  at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 398
  --- End of inner exception stack trace ---
  at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 402
  at System.Net.Http.Http3RequestStream.SendContentAsync(HttpContent content, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 399
  at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 186
  at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 320
  at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 245
  at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 257
  at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1006
  at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1050
  at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
  at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 530
  at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 95
  at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in C:\app\ClientOperations.cs:line 369
2023-03-20T13:37:38.9622574Z httpstress-client-1  | --- End of stack trace from previous location ---
  at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204

SslStress

2 x #76044 on Windows, which is less frequent than earlier.

Mar 28 '23 15:03 antonfirsov

Report 2023/04/25

There are massive build failures #84800 and #85338 polluting the logs, it would be better to produce a report when at least one of those issues is fixed.

Apr 25 '23 16:04 antonfirsov

runtime runtime copied to clipboard