ice icon indicating copy to clipboard operation
ice copied to clipboard
verified publisher icon zeroc-ice

Sporaidc csharp/Ice/adapterDeactivation test failure

Open externl opened this issue 1 year ago • 1 comments

This test started failing after I updated the ci workflow to build and run all of the languages and tests in one step. The test fails sporadically but always at the same place and with the same configuration

ws,compress,ipv6,serialize,mx testing object adapter with router...

The exception causing the error seems to be:

Caused by: System.Net.Sockets.SocketException: Connection refused
Unhandled exception. System.ArgumentException: The supplied System.Net.SocketAddress is an invalid size for the System.Net.IPEndPoint end point. (Parameter 'socketAddress')

*** [292/358] Running csharp/Ice/adapterDeactivation tests ***
[ running client/server test - 04/19/24 14:49:53 ]
- Config: ssl
(dotnet /Users/runner/work/ice/ice/csharp/test/Ice/adapterDeactivation/msbuild/server/net8.0/server.dll --Ice.Default.Host=127.0.0.1 --Test.BasePort=14000 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Plugin.IceSSL= --IceSSL.*** --IceSSL.DefaultDir=/Users/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.VerifyPeer=2 --IceSSL.CertFile=server.p12 --Ice.ThreadPool.Server.Size=1 --Ice.ThreadPool.Server.SizeMax=3 --Ice.ThreadPool.Server.SizeWarn=0 --Ice.PrintAdapterReady=1)
(dotnet /Users/runner/work/ice/ice/csharp/test/Ice/adapterDeactivation/msbuild/client/net8.0/client.dll --Ice.Default.Host=127.0.0.1 --Test.BasePort=14000 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Plugin.IceSSL= --IceSSL.*** --IceSSL.DefaultDir=/Users/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.VerifyPeer=2 --IceSSL.CertFile=client.p12)
testing stringToProxy... ok
testing checked cast... ok
creating/destroying/recreating object adapter... ok
creating/activating/deactivating object adapter in one operation... ok
testing connection closure... ok
testing object adapter published endpoints... ok
testing object adapter with bi-dir connection... ok
testing object adapter with router... ok
testing object adapter creation with port in use... ok
deactivating object adapter in the server... ok
testing whether server is gone... ok
testing server idle time...ok
[ running client/server test - 04/19/24 14:49:57 ]
- Config: ws,compress,ipv6,serialize,mx
(dotnet /Users/runner/work/ice/ice/csharp/test/Ice/adapterDeactivation/msbuild/server/net8.0/server.dll --Ice.Default.Host=::1 --Test.BasePort=14000 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ws --Ice.Override.Compress=1 --Ice.ThreadPool.Server.Serialize=1 --Ice.IPv6=1 --Ice.PreferIPv6Address=1 --Ice.Admin.Endpoints="tcp -h \"::1\"" --Ice.Admin.InstanceName=Server --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --Ice.ThreadPool.Server.Size=1 --Ice.ThreadPool.Server.SizeMax=3 --Ice.ThreadPool.Server.SizeWarn=0 --Ice.PrintAdapterReady=1)
(dotnet /Users/runner/work/ice/ice/csharp/test/Ice/adapterDeactivation/msbuild/client/net8.0/client.dll --Ice.Default.Host=::1 --Test.BasePort=14000 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ws --Ice.Override.Compress=1 --Ice.ThreadPool.Server.Serialize=1 --Ice.IPv6=1 --Ice.PreferIPv6Address=1 --Ice.Admin.Endpoints="tcp -h \"::1\"" --Ice.Admin.InstanceName=Client --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none)
testing stringToProxy... ok
testing checked cast... ok
creating/destroying/recreating object adapter... ok
creating/activating/deactivating object adapter in one operation... ok
testing connection closure... ok
testing object adapter published endpoints... ok
testing object adapter with bi-dir connection... ok
testing object adapter with router... -! 4/19/2024 14:49:58:171 client: warning: connection exception:
   Ice.ConnectionLostException
       error = 0
      at IceInternal.StreamSocket.finishRead(Buffer buf) in /Users/runner/work/ice/ice/csharp/src/Ice/StreamSocket.cs:line 205
      at IceInternal.TcpTransceiver.finishRead(Buffer buf) in /Users/runner/work/ice/ice/csharp/src/Ice/TcpTransceiver.cs:line 61
      at IceInternal.WSTransceiver.finishRead(Buffer buf) in /Users/runner/work/ice/ice/csharp/src/Ice/WSTransceiver.cs:line 558
      at Ice.ConnectionI.finishAsync(Int32 operation) in /Users/runner/work/ice/ice/csharp/src/Ice/ConnectionI.cs:line 987
   local address = ::1:52005
   remote address = ::1:14000
Ice.ConnectionRefusedException
    error = 0
   at Ice.RouterPrxHelper.getClientProxy(Optional`1& hasRoutingTable, OptionalContext context) in /Users/runner/work/ice/ice/csharp/src/Ice/generated/Router.cs:line 282
   at IceInternal.RouterInfo.getClientEndpoints() in /Users/runner/work/ice/ice/csharp/src/Ice/RouterInfo.cs:line 73
   at IceInternal.OutgoingConnectionFactory.setRouterInfo(RouterInfo routerInfo) in /Users/runner/work/ice/ice/csharp/src/Ice/ConnectionFactory.cs:line 178
   at Ice.ObjectAdapterI..ctor(Instance instance, Communicator communicator, ObjectAdapterFactory objectAdapterFactory, String name, RouterPrx router, Boolean noConfig, SslServerAuthenticationOptions serverAuthenticationOptions) in /Users/runner/work/ice/ice/csharp/src/Ice/ObjectAdapterI.cs:line 955
   at IceInternal.ObjectAdapterFactory.createObjectAdapter(String name, RouterPrx router, SslServerAuthenticationOptions serverAuthenticationOptions) in /Users/runner/work/ice/ice/csharp/src/Ice/ObjectAdapterFactory.cs:line 172
   at Ice.CommunicatorI.createObjectAdapterWithRouter(String name, RouterPrx router) in /Users/runner/work/ice/ice/csharp/src/Ice/CommunicatorI.cs:line 108
   at Ice.adapterDeactivation.AllTests.allTests(TestHelper helper) in /Users/runner/work/ice/ice/csharp/test/Ice/adapterDeactivation/AllTests.cs:line 124
   at Ice.adapterDeactivation.Client.runAsync(String[] args) in /Users/runner/work/ice/ice/csharp/test/Ice/adapterDeactivation/Client.cs:line 14
   at Test.TestDriver.runTestAsync[T](String[] args) in /Users/runner/work/ice/ice/csharp/test/TestCommon/TestHelper.cs:line 232
Caused by: System.Net.Sockets.SocketException: Connection refused
Unhandled exception. System.ArgumentException: The supplied System.Net.SocketAddress is an invalid size for the System.Net.IPEndPoint end point. (Parameter 'socketAddress')
   at System.Net.IPEndPoint.Create(SocketAddress socketAddress)
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAccept(SocketAddress remoteSocketAddress)
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSyncSuccess(Int32 bytesTransferred, SocketFlags flags)
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags)
   at System.Net.Sockets.SocketAsyncEventArgs.AcceptCompletionCallback(IntPtr acceptedFileDescriptor, Memory`1 socketAddress, SocketError socketError)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

externl avatar Apr 19 '24 15:04 externl

I can reproduce this locally, but it seems to only happen when my system is under load.

To encourage the crash, I've been using stress:

brew install stress
stress --cpu 10

Also maybe relevant: https://github.com/dotnet/runtime/issues/40913

externl avatar Apr 24 '24 18:04 externl

The underlying cause is likely this .NET bug https://github.com/dotnet/runtime/issues/102663, see also #2677

The unhandled exception issue was fixed in #2704

pepone avatar Sep 05 '24 12:09 pepone