runtime icon indicating copy to clipboard operation
runtime copied to clipboard

Published 20 hours ago • verified publisher icon dotnet

Failures in SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3 test

Open jkotas opened this issue 3 years ago • 7 comments

Hit in #74885

Full log https://dev.azure.com/dnceng/public/_build/results?buildId=1979617&view=ms.vss-test-web.build-test-results-tab&runId=50623688&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=203530

System.AggregateException : One or more errors occurred. (Assert.Contains() Failure
Not found: 1024
In value:  System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.Net.Quic.QuicException: An internal error has occurred. Status code: QUIC_STATUS_SUCCESS
at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 221
at System.Net.Quic.QuicStream.HandleEventShutdownComplete(SHUTDOWN_COMPLETE_e__Struct& data) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 595
at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 632
at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 656
--- End of stack trace from previous location ---
at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 235
at System.Net.Quic.QuicStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 287 at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 788 at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 337 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206 --- End of inner exception stack trace --- at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 308 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 237 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 251 at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 990 at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1034 at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30 at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 532 at Xunit.Assert.RecordExceptionAsync(Func1 testCode) in //src/xunit.assert/Asserts/Record.cs:line 111) (An internal error has occurred. Status code: QUIC_STATUS_SUCCESS)

{ "ErrorMessage":"System.Net.Quic.QuicException: An internal error has occurred. Status code: QUIC_STATUS_SUCCESS" }

Report

Build Definition Test
12403 dotnet/runtime System.Net.Http.Functional.Tests.WorkItemExecution
6270 dotnet/runtime System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.ServerClosesOutboundControlStream_ClientClosesConnection
6262 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3.LargeSingleHeader_ThrowsException
6233 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3.LargeSingleHeader_ThrowsException
5107 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3.LargeSingleHeader_ThrowsException
3704 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3.LargeSingleHeader_ThrowsException
1979617 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3.LargeSingleHeader_ThrowsException
1977906 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3.LargeSingleHeader_ThrowsException

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 8

jkotas avatar Sep 01 '22 00:09 jkotas

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details

Hit in #74885

Full log https://dev.azure.com/dnceng/public/_build/results?buildId=1979617&view=ms.vss-test-web.build-test-results-tab&runId=50623688&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=203530

System.AggregateException : One or more errors occurred. (Assert.Contains() Failure
Not found: 1024
In value:  System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.Net.Quic.QuicException: An internal error has occurred. Status code: QUIC_STATUS_SUCCESS
at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 221
at System.Net.Quic.QuicStream.HandleEventShutdownComplete(SHUTDOWN_COMPLETE_e__Struct& data) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 595
at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 632
at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 656
--- End of stack trace from previous location ---
at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 235
at System.Net.Quic.QuicStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 287 at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 788 at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 337 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206 --- End of inner exception stack trace --- at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 308 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 237 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 251 at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 990 at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1034 at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30 at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 532 at Xunit.Assert.RecordExceptionAsync(Func1 testCode) in //src/xunit.assert/Asserts/Record.cs:line 111) (An internal error has occurred. Status code: QUIC_STATUS_SUCCESS)
Author: jkotas
Assignees: -
Labels:

area-System.Net.Http, untriaged
Milestone: -

ghost avatar Sep 01 '22 00:09 ghost

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details

Hit in #74885

Full log https://dev.azure.com/dnceng/public/_build/results?buildId=1979617&view=ms.vss-test-web.build-test-results-tab&runId=50623688&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=203530

System.AggregateException : One or more errors occurred. (Assert.Contains() Failure
Not found: 1024
In value:  System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.Net.Quic.QuicException: An internal error has occurred. Status code: QUIC_STATUS_SUCCESS
at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 221
at System.Net.Quic.QuicStream.HandleEventShutdownComplete(SHUTDOWN_COMPLETE_e__Struct& data) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 595
at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 632
at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 656
--- End of stack trace from previous location ---
at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 235
at System.Net.Quic.QuicStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 287 at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 788 at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 337 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206 --- End of inner exception stack trace --- at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 308 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 237 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 251 at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 990 at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1034 at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30 at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 532 at Xunit.Assert.RecordExceptionAsync(Func1 testCode) in //src/xunit.assert/Asserts/Record.cs:line 111) (An internal error has occurred. Status code: QUIC_STATUS_SUCCESS)
Author: jkotas
Assignees: -
Labels:

area-System.Net.Http, area-System.Net.Quic
Milestone: -

ghost avatar Sep 01 '22 00:09 ghost

may be fixed by #74634 (and https://github.com/dotnet/runtime/pull/74611 it depends on) cc: @rzikm

In either case it feels strange to throw with QUIC_STATUS_SUCCESS....

wfurt avatar Sep 01 '22 00:09 wfurt

Yes, the thing is... this is the code we receive from MsQuic... so if this continues after we merge the two PRs you mention (and we rule out any memory corruption done by us), we would need to involve MsQuic team.

rzikm avatar Sep 01 '22 11:09 rzikm

Also hit in #74669 (log) and #74645 (log) with a bit different error:

System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3.LargeSingleHeader_ThrowsException(maxResponseHeadersLength: 1) [FAIL]
      System.AggregateException : One or more errors occurred. (Assert.Contains() Failure
      Not found: 1024
      In value:  System.Net.Http.HttpRequestException: An error occurred while sending the request.
                  ---> System.Net.Quic.QuicException: The connection timed out from inactivity.

I think we've already seen this type of failure before, when @MihaZupan just added it?

PR to disable: https://github.com/dotnet/runtime/pull/75149

CarnaViire avatar Sep 06 '22 18:09 CarnaViire

Yes, this test was already disabled in https://github.com/dotnet/runtime/pull/73945 and then re-enabled in https://github.com/dotnet/runtime/pull/74479. The fact that it's still failing indicates a QUIC/HTTP3 issue.

MihaZupan avatar Sep 06 '22 18:09 MihaZupan

@MihaZupan I meant specifically a "timeout from inactivity". Because there were a bunch of different failures as well

CarnaViire avatar Sep 06 '22 18:09 CarnaViire

No hit in a month, last comments from sept, closing.

ManickaP avatar Jun 01 '23 10:06 ManickaP

The test disabled against this issue needs to be re-enabled before closing: https://github.com/dotnet/runtime/blob/7bcd5091ef7df1444de7c7e1dfc3fd281e773fff/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs#L1607

jkotas avatar Jun 01 '23 13:06 jkotas

The test disabled against this issue needs to be re-enabled before closing:

https://github.com/dotnet/runtime/blob/7bcd5091ef7df1444de7c7e1dfc3fd281e773fff/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs#L1607

:facepalm: Thanks for catching that, I completely overlooked it.

ManickaP avatar Jun 01 '23 14:06 ManickaP

The test was re-enabled in PR #91629 on 9/6.

The same test started failing with different error on 9/7 - reported in #91757 The test was disabled against THIS issue in PR #91930 on 9/12. That was an accident -- we should disable it against the new issue #91757 which captures latest failure.

@liveans please change the test number against which it is disabled, then let's close this issue as dupe of #91757 (while removing the disabled-test label). Let's keep it in 9.0.

karelz avatar Oct 11 '23 09:10 karelz

I changed the issue number on the test, closing this issue as a duplicate of #91757, and deleting the disabled-test label.

liveans avatar Oct 11 '23 11:10 liveans

Duplicate of #91757

karelz avatar Oct 11 '23 12:10 karelz