runtime icon indicating copy to clipboard operation
runtime copied to clipboard

[wasm] WBT: Blazor tests on windows failing with `Failed to bind to address` for random ports

Open radical opened this issue 2 years ago • 7 comments

Reproduction Steps

  • Queue: browser-wasm linux Release WasmBuildTests
  • Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=413445&view=results
  • Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-heads-release-80-rc1-e5e75e14b32149239e/NoWorkload-Wasm.Build.Tests/1/console.cabf329c.log?helixlogtype=result
  • Output:
    Wasm.Build.Tests.Blazor.WorkloadRequiredTests.WorkloadNotRequiredForInvariantGlobalization(config: "Release", invariant: False, publish: False) [FAIL]
       Expected 0 exit code but got 134: /root/helix/work/workitem/e/dotnet-none/dotnet run -c Release --no-build
      Standard Output:
      [] fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      []       Hosting failed to start
      []       System.IO.IOException: Failed to bind to address http://127.0.0.1:5145: address already in use.
      []        ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use
      []        ---> System.Net.Sockets.SocketException (98): Address already in use
      []          at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, Boolean disconnectOnFailure, String callerName)
      []          at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
      []          at System.Net.Sockets.Socket.Bind(EndPoint localEP)
      []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
      []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
      []          --- End of inner exception stack trace ---
      []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
      []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
      []       --- End of stack trace from previous location ---
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
      []          --- End of inner exception stack trace ---
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(ListenOptions[] listenOptions, AddressBindContext context, Func`2 useHttps, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
      []          at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
      []          at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
      [] Unhandled exception. System.IO.IOException: Failed to bind to address http://127.0.0.1:5145: address already in use.
      []  ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use
      []  ---> System.Net.Sockets.SocketException (98): Address already in use
      []    at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, Boolean disconnectOnFailure, String callerName)
      []    at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
      []    at System.Net.Sockets.Socket.Bind(EndPoint localEP)
      []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
      []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
      []    --- End of inner exception stack trace ---
      []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
      []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
      [] --- End of stack trace from previous location ---
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
      []    --- End of inner exception stack trace ---
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(ListenOptions[] listenOptions, AddressBindContext context, Func`2 useHttps, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
      []    at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
      []    at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
      []    at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
      []    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
      []    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
      []    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
      []    at Microsoft.AspNetCore.Components.WebAssembly.DevServer.Program.Main(String[] args)
      

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "Microsoft.AspNetCore.Connections.AddressInUseException:",
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: Result validation: :warning: Build internal information not found. This may happen if your build is too old. Please use a build that is no older than two weeks. If the problem persists, contact .NET Engineering Services Team and share this issue. Validation performed at: 10/17/2023 1:46:13 AM UTC

Report

Build Definition Test Pull Request
524779 dotnet/runtime Workloads-Wasm.Build.Tests.WasmTemplateTests.WorkItemExecution dotnet/runtime#96459
519412 dotnet/runtime Wasm.Build.Tests.Blazor.BuildPublishTests.DefaultTemplate_WithoutWorkload dotnet/runtime#96677
508242 dotnet/runtime Wasm.Build.Tests.Blazor.BuildPublishTests.DefaultTemplate_WithoutWorkload dotnet/runtime#96177
507454 dotnet/runtime Wasm.Build.Tests.Blazor.WorkloadRequiredTests.WorkloadNotRequiredForInvariantGlobalization dotnet/runtime#96177

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 4

radical avatar Oct 02 '23 21:10 radical

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

Tagging this for KnownBuildError in case it's happening consistently and we need to fix this in 8.0.

{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "Wasm\.Build\.Tests.*System\.IO\.IOException: Failed to bind to address http://127\.0\.0\.1:(?!5000)",
  "ExcludeConsoleLog": false
}

Reproduction Steps

  • Queue: browser-wasm linux Release WasmBuildTests
  • Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=413445&view=results
  • Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-heads-release-80-rc1-e5e75e14b32149239e/NoWorkload-Wasm.Build.Tests/1/console.cabf329c.log?helixlogtype=result

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}
Author: radical
Assignees: -
Labels:

arch-wasm, blocking-clean-ci, Known Build Error

Milestone: -

ghost avatar Oct 02 '23 21:10 ghost

@missymessa Any idea why this is not getting picked up for known-build-error analysis?

radical avatar Oct 02 '23 22:10 radical

@maraf Looking at the logs, the same port number is used in one of the earlier tests, but even if that process is still running the devserver shouldn't have picked a port still in use. Seems like a devserver issue?

radical avatar Oct 02 '23 23:10 radical

@radical it wasn't picked up because it had escaping problems, you can notice this because the json message was in red, like this:

{
 "ErrorPattern": "Wasm\.Build\.Tests.*System\.IO\.IOException: Failed to bind to address http://127\.0\.0\.1:(?!5000)"
}

I fixed the escaping and now the validation was able to run, but I noticed that your regex was trying to look into multiple lines, this feature is not available for known issues, the error needs to be a single line, also the validation returned a backtracking error, all this needs to be fixed in order than known issues can work

let me know if you need help with any of this

AlitzelMendez avatar Oct 03 '23 00:10 AlitzelMendez

I fixed the escaping and now the validation was able to run, but I noticed that your regex was trying to look into multiple lines, this feature is not available for known issues,

Oh, I was basing this on the doc:

https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssues.md#regex-matching

We recommend you test your regular expression, to do it you can use [regex101 tester](https://regex101.com/) (choose .NET (C#) flavor) with the following regex options:

Single line
Insentitive
No backtracking

The single line here would mean that . would match newlines also, even according to that website.

If that isn't supported then what is the recommended way to identify failure messages that are not unique. For example System.IO.IOException: Failed to bind to address can match lots of test failures but the issue is specifically about this failure in Wasm.Build.Tests. Without the multi-line regex, how do I narrow down the match?

radical avatar Oct 03 '23 00:10 radical

If that isn't supported then what is the recommended way to identify failure messages that are not unique. For example System.IO.IOException: Failed to bind to address can match lots of test failures but the issue is specifically about this failure in Wasm.Build.Tests. Without the multi-line regex, how do I narrow down the match?

Unfortunately, there is not a lot that we can suggest on these cases, if this is narrow to a specific test name you can use the test name, otherwise the only option is try to identify an error line that is unique to your issue.

we provide a little bit more information about why the multiline is not supported of this issue: https://github.com/dotnet/arcade/issues/13253

AlitzelMendez avatar Oct 03 '23 22:10 AlitzelMendez

If that isn't supported then what is the recommended way to identify failure messages that are not unique. For example System.IO.IOException: Failed to bind to address can match lots of test failures but the issue is specifically about this failure in Wasm.Build.Tests. Without the multi-line regex, how do I narrow down the match?

Unfortunately, there is not a lot that we can suggest on these cases, if this is narrow to a specific test name you can use the test name, otherwise the only option is try to identify an error line that is unique to your issue.

we provide a little bit more information about why the multiline is not supported of this issue: dotnet/arcade#13253

Maybe allowing matching against the helix item name, or maybe two single line patterns to match? That should be cheaper than a multiline regex search over a big text file.

radical avatar Oct 03 '23 23:10 radical

does it not happen anymore ? Or the filter doesn't work ?

pavelsavara avatar Apr 25 '24 15:04 pavelsavara