ice icon indicating copy to clipboard operation
ice copied to clipboard
verified publisher icon zeroc-ice

csharp/IceGrid/simple failure on Ubuntu 24.04

Open bernardnormier opened this issue 1 year ago • 1 comments

From https://github.com/zeroc-ice/ice/actions/runs/11075786900/job/30777485537 (on main):

*** [67/304] Running csharp/IceGrid/simple tests ***
[ running without deployment test - 09/27/24 18:49:33 ]
- Config: ssl,mx
starting IceGrid registry Master... (/home/runner/work/ice/ice/cpp/bin/icegridregistry --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=0 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Server --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=server.p12 --Ice.PrintStackTraces=1 --IceGrid.InstanceName=TestIceGrid --IceGrid.Registry.PermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.AdminPermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.SSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.AdminSSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.Server.Endpoints=default --IceGrid.Registry.Internal.Endpoints=default --IceGrid.Registry.Client.Endpoints="default -p 14020" --IceGrid.Registry.Discovery.Port=14099 --IceGrid.Registry.SessionManager.Endpoints=default --IceGrid.Registry.AdminSessionManager.Endpoints=default --IceGrid.Registry.ReplicaName=Master --Ice.ProgramName=Master --Ice.PrintAdapterReady=1 --Ice.ThreadPool.Client.SizeWarn=0 --IceGrid.Registry.LMDB.MapSize=1 --IceGrid.Registry.LMDB.Path=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/registry-Master --IceGrid.Registry.Client.ThreadPool.SizeWarn=0 --IceGrid.Registry.DefaultTemplates="/home/runner/work/ice/ice/cpp/config/templates.xml" --IceGrid.Registry.DynamicRegistration=1 --IceGrid.Registry.Trace.Discovery=2 --Ice.LogFile=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/Master-092724-1849.log env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
ok
starting IceGrid registry Slave1... (/home/runner/work/ice/ice/cpp/bin/icegridregistry --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=0 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Server --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=server.p12 --Ice.PrintStackTraces=1 --IceGrid.InstanceName=TestIceGrid --IceGrid.Registry.PermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.AdminPermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.SSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.AdminSSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.Server.Endpoints=default --IceGrid.Registry.Internal.Endpoints=default --IceGrid.Registry.Client.Endpoints="default -p 14021" --IceGrid.Registry.Discovery.Port=14099 --IceGrid.Registry.SessionManager.Endpoints=default --IceGrid.Registry.AdminSessionManager.Endpoints=default --IceGrid.Registry.ReplicaName=Slave1 --Ice.ProgramName=Slave1 --Ice.PrintAdapterReady=1 --Ice.ThreadPool.Client.SizeWarn=0 --IceGrid.Registry.LMDB.MapSize=1 --IceGrid.Registry.LMDB.Path=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/registry-Slave1 --IceGrid.Registry.Client.ThreadPool.SizeWarn=0 --IceGrid.Registry.DefaultTemplates="/home/runner/work/ice/ice/cpp/config/templates.xml" --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --IceGrid.Registry.DynamicRegistration=1 --IceGrid.Registry.Trace.Discovery=2 --Ice.LogFile=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/Slave1-092724-1849.log env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
ok
starting IceGrid registry Slave2... (/home/runner/work/ice/ice/cpp/bin/icegridregistry --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=0 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Server --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=server.p12 --Ice.PrintStackTraces=1 --IceGrid.InstanceName=TestIceGrid --IceGrid.Registry.PermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.AdminPermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.SSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.AdminSSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.Server.Endpoints=default --IceGrid.Registry.Internal.Endpoints=default --IceGrid.Registry.Client.Endpoints="default -p 14022" --IceGrid.Registry.Discovery.Port=14099 --IceGrid.Registry.SessionManager.Endpoints=default --IceGrid.Registry.AdminSessionManager.Endpoints=default --IceGrid.Registry.ReplicaName=Slave2 --Ice.ProgramName=Slave2 --Ice.PrintAdapterReady=1 --Ice.ThreadPool.Client.SizeWarn=0 --IceGrid.Registry.LMDB.MapSize=1 --IceGrid.Registry.LMDB.Path=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/registry-Slave2 --IceGrid.Registry.Client.ThreadPool.SizeWarn=0 --IceGrid.Registry.DefaultTemplates="/home/runner/work/ice/ice/cpp/config/templates.xml" --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --IceGrid.Registry.DynamicRegistration=1 --IceGrid.Registry.Trace.Discovery=2 --Ice.LogFile=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/Slave2-092724-1849.log env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
ok
starting IceGrid node localnode... (/home/runner/work/ice/ice/cpp/bin/icegridnode --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Server --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=server.p12 --Ice.PrintStackTraces=1 --IceGrid.InstanceName=TestIceGrid --IceGrid.Node.Endpoints=default --IceGrid.Node.WaitTime=240 --Ice.ProgramName=icegridnode --IceGrid.Node.Trace.Replica=0 --IceGrid.Node.Trace.Activator=0 --IceGrid.Node.Trace.Adapter=0 --IceGrid.Node.Trace.Server=0 --IceGrid.Node.ThreadPool.SizeWarn=0 --IceGrid.Node.PrintServersReady=node --IceGrid.Node.Name=localnode --IceGrid.Node.Data=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/node-localnode --IceGrid.Node.PropertiesOverride="Ice.Default.Host=127.0.0.1 Test.BasePort=14000 Ice.Warn.Connections=1 Ice.Default.Protocol=ssl Ice.IPv6=0 Ice.Admin.Endpoints=\"tcp -h 127.0.0.1\" Ice.Admin.InstanceName=Server IceMX.Metrics.Debug.GroupBy=id IceMX.Metrics.Parent.GroupBy=parent IceMX.Metrics.All.GroupBy=none IceSSL.*** IceSSL.DefaultDir=/home/runner/work/ice/ice/certs IceSSL.CAs=cacert.pem IceSSL.VerifyPeer=2 IceSSL.CertFile=server.p12 Ice.ThreadPool.Server.Size=1 Ice.ThreadPool.Server.SizeMax=3 Ice.ThreadPool.Server.SizeWarn=0 Ice.PrintAdapterReady=1" --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020:default -p 14021:default -p 14022" env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
ok
(dotnet /home/runner/work/ice/ice/csharp/test/IceGrid/simple/msbuild/server/net8.0/server.dll --Ice.Default.Host=127.0.0.1 --Test.BasePort=14000 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Server --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.VerifyPeer=2 --IceSSL.CertFile=server.p12 --Ice.ThreadPool.Server.Size=1 --Ice.ThreadPool.Server.SizeMax=3 --Ice.ThreadPool.Server.SizeWarn=0 --Ice.PrintAdapterReady=1 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --TestAdapter.Endpoints=default --TestAdapter.AdapterId=TestAdapter)
(dotnet /home/runner/work/ice/ice/csharp/test/IceGrid/simple/msbuild/client/net8.0/client.dll --Ice.Default.Host=127.0.0.1 --Test.BasePort=14000 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Client --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.VerifyPeer=2 --IceSSL.CertFile=client.p12 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --IceLocatorDiscovery.Timeout=50 --IceLocatorDiscovery.RetryCount=5 --IceLocatorDiscovery.Interface= --IceLocatorDiscovery.Port=14099 --IceLocatorDiscovery.Trace.Lookup=3 --Ice.LogFile=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/client-092724-1849.log)
testing stringToProxy... ok
testing IceGrid.Locator is present... ok
testing checked cast... ok
pinging server... ok
testing locator finder... ok
testing discovery... failed:
saved /home/runner/work/ice/ice/csharp/test/IceGrid/simple/client-092724-1849.log
-! 09/27/2024 18:49:36:754 server: warning: connection exception:
   Ice.ConnectionLostException: Exception of type 'Ice.ConnectionLostException' was thrown.
      at Ice.SSL.TransceiverI.finishRead(Buffer buf) in /home/runner/work/ice/ice/csharp/src/Ice/SSL/TransceiverI.cs:line 176
      at Ice.Internal.IdleTimeoutTransceiverDecorator.finishRead(Buffer buf) in /home/runner/work/ice/ice/csharp/src/Ice/Internal/IdleTimeoutTransceiverDecorator.cs:line 56
      at Ice.ConnectionI.finishAsync(Int32 operation) in /home/runner/work/ice/ice/csharp/src/Ice/ConnectionI.cs:line 716
   local address = 127.0.0.1:33053
   remote address = 127.0.0.1:58056
(/home/runner/work/ice/ice/cpp/bin/icegridadmin --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Client --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=client.p12 --Ice.PrintStackTraces=1 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --IceGridAdmin.Username=admin1 --IceGridAdmin.*** -r Master -e "node shutdown localnode" env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
(/home/runner/work/ice/ice/cpp/bin/icegridadmin --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Client --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=client.p12 --Ice.PrintStackTraces=1 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --IceGridAdmin.Username=admin1 --IceGridAdmin.*** -r Master -e "registry shutdown Master" env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
(/home/runner/work/ice/ice/cpp/bin/icegridadmin --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Client --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=client.p12 --Ice.PrintStackTraces=1 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14021" --IceGridAdmin.Username=admin1 --IceGridAdmin.*** -r Slave1 -e "registry shutdown Slave1" env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
(/home/runner/work/ice/ice/cpp/bin/icegridadmin --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Client --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=client.p12 --Ice.PrintStackTraces=1 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14022" --IceGridAdmin.Username=admin1 --IceGridAdmin.*** -r Slave2 -e "registry shutdown Slave2" env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
saved /home/runner/work/ice/ice/csharp/test/IceGrid/simple/Slave2-092724-1849.log
saved /home/runner/work/ice/ice/csharp/test/IceGrid/simple/Slave1-092724-1849.log
saved /home/runner/work/ice/ice/csharp/test/IceGrid/simple/Master-092724-1849.log

test in IceGrid/simple failed:

bernardnormier avatar Sep 27 '24 20:09 bernardnormier

Still failing: https://github.com/zeroc-ice/ice/actions/runs/11387205780/job/31681241615?pr=2916

bernardnormier avatar Oct 17 '24 17:10 bernardnormier

The client log shows and exception sending the datagram lookup requets

-- 10/17/2024 15:04:03:894 client: Lookup: retrying locator lookup:
   lookup = IceLocatorDiscovery/Lookup -d -e 1.1:udp -h 239.255.0.1 -p 14299 --interface 127.0.0.1:udp -h 239.255.0.1 -p 14299 --interface 10.1.0.17:udp -h 239.255.0.1 -p 14299 --interface 172.17.0.1
   retry count = 1
   instance name = unknown
-- 10/17/2024 15:04:03:997 client: Lookup: locator lookup timed out:
   lookup = IceLocatorDiscovery/Lookup -d -e 1.1:udp -h 239.255.0.1 -p 14299 --interface 127.0.0.1:udp -h 239.255.0.1 -p 14299 --interface 10.1.0.17:udp -h 239.255.0.1 -p 14299 --interface 172.17.0.1
   instance name = unknown
-- 10/17/2024 15:04:04:040 client: Lookup: looking up locator:
   lookup = IceLocatorDiscovery/Lookup -d -e 1.1:udp -h 239.255.0.1 -p 0 --interface unknown
-! 10/17/2024 15:04:04:058 client: warning: failed to lookup locator with lookup proxy `IceLocatorDiscovery/Lookup -d -e 1.1:udp -h 239.255.0.1 -p 0 --interface unknown':
   Ice.SocketException: Exception of type 'Ice.SocketException' was thrown.
    ---> System.ArgumentException: couldn't find interface `unknown'
      at Ice.Internal.Network.getInterfaceAddress(String iface, AddressFamily family) in /home/runner/work/ice/ice/csharp/src/Ice/Internal/Network.cs:line 1170
      at Ice.Internal.Network.setMcastInterface(Socket socket, String iface, AddressFamily family) in /home/runner/work/ice/ice/csharp/src/Ice/Internal/Network.cs:line 392
      --- End of inner exception stack trace ---
      at Ice.Internal.OutgoingAsyncBase.invokeException() in /home/runner/work/ice/ice/csharp/src/Ice/Internal/OutgoingAsync.cs:line 100
   --- End of stack trace from previous location ---
      at IceLocatorDiscovery.LocatorI.<invoke>g__preformFindLocatorAsync|5_0(LookupPrx lookupPrx, LookupReplyPrx lookupReplyPrx) in /home/runner/work/ice/ice/csharp/src/IceLocatorDiscovery/PluginI.cs:line 465
-- 10/17/2024 15:04:04:059 client: Lookup: locator lookup failed:
   lookup = IceLocatorDiscovery/Lookup -d -e 1.1:udp -h 239.255.0.1 -p 0 --interface unknown
   Ice.SocketException: Exception of type 'Ice.SocketException' was thrown.
    ---> System.ArgumentException: couldn't find interface `unknown'
      at Ice.Internal.Network.getInterfaceAddress(String iface, AddressFamily family) in /home/runner/work/ice/ice/csharp/src/Ice/Internal/Network.cs:line 1170
      at Ice.Internal.Network.setMcastInterface(Socket socket, String iface, AddressFamily family) in /home/runner/work/ice/ice/csharp/src/Ice/Internal/Network.cs:line 392
      --- End of inner exception stack trace ---
      at Ice.Internal.OutgoingAsyncBase.invokeException() in /home/runner/work/ice/ice/csharp/src/Ice/Internal/OutgoingAsync.cs:line 100
   --- End of stack trace from previous location ---
      at IceLocatorDiscovery.LocatorI.<invoke>g__preformFindLocatorAsync|5_0(LookupPrx lookupPrx, Looku

We actually set the unknown interface in the test.

pepone avatar Nov 01 '24 17:11 pepone

Currently waiting for a new failure with more tracing. I can't reproduce it in a devcontainer.

bernardnormier avatar Nov 05 '24 22:11 bernardnormier

I just got this in a PR: https://github.com/zeroc-ice/ice/actions/runs/11784745850/job/32824557772

test-logs-debug-ubuntu-24.04.zip

externl avatar Nov 11 '24 20:11 externl

This isn't specific to Ubuntu apparently, and doesn't require ssl, mx. I got this when testing locally on Windows 11, with normal tcp:

*** [137/256] Running csharp/IceGrid/simple tests ***
[ running without deployment test - 11/18/24 16:21:34 ]
- Config: Debug,x64
starting IceGrid registry Master... ok
starting IceGrid registry Slave1... ok
starting IceGrid registry Slave2... ok
starting IceGrid node localnode... ok
testing stringToProxy... ok
testing IceGrid.Locator is present... ok
testing checked cast... ok
pinging server... ok
testing locator finder... ok
testing discovery... failed:


saved D:\Code\Workspace\ice\csharp\test\IceGrid\simple\client-111824-1621.log
saved D:\Code\Workspace\ice\csharp\test\IceGrid\simple\server-111824-1621.log
saved D:\Code\Workspace\ice\csharp\test\IceGrid\simple\Slave2-111824-1621.log
saved D:\Code\Workspace\ice\csharp\test\IceGrid\simple\Slave1-111824-1621.log
saved D:\Code\Workspace\ice\csharp\test\IceGrid\simple\Master-111824-1621.log
unexpected exit status: expected: 0, got 1


test in IceGrid/simple failed:
unexpected exit status: expected: 0, got 1

Here are my logs: IceGrid-Simple-Failure-Logs.zip

InsertCreativityHere avatar Nov 18 '24 21:11 InsertCreativityHere

This failure is very difficult to reproduce and was probably fixed by #3177.

bernardnormier avatar Nov 22 '24 14:11 bernardnormier