Tests failures: System.Net.NameResolution.Tests / DnsObsolete* & Dns_GetHostEntry*

[ghost]

Type of failures

Affected tests:

• System.Net.NameResolution.Tests.GetHostByNameTest (both always fail together)
  • DnsObsoleteGetHostByName_EmptyString_ReturnsHostName
  • DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName
• System.Net.NameResolution.Tests.GetHostEntryTest (both always fail together)
  • Dns_GetHostEntry_HostString_Ok
  • Dns_GetHostEntryAsync_HostString_Ok

Failure text:

• OSX: Device not configured
• Linux: No such device or address

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Device not configured
OR
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : No such device or address

at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6) in /Users/buildagent/agent/_work/34/s/corefx/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 46
at System.Net.Dns.GetHostByName(String hostName) in /Users/buildagent/agent/_work/34/s/corefx/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 41
at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /Users/buildagent/agent/_work/34/s/corefx/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 108

Note:

• Dns_GetHostEntryAsync_HostString_Ok fails almost always with same error twice, wrapped in System.AggregateException.

Older failure on OSX (prior to 2017/9/15)

Assert.Contains() Failure
Not found: DCI-Mac-Build-068.local
In value:  dci-mac-build-068.local
at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /Users/buildagent/agent/_work/30/s/corefx/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 108

History of failures

Each test couple always fails or passes - maybe bad machine setup?

Day	Build	OS	Test
9/6	20170906.01	OSX10.12	DnsObsolete*
9/10	20170910.01	Ubuntu14.04	DnsObsolete* & Dns_GetHostEntry*
9/13	20170913.02	OSX10.12	DnsObsolete*
9/15	20170915.01	OSX10.12	DnsObsolete*
9/20	20170920.03	Debian90	DnsObsolete* & Dns_GetHostEntry*
9/30	20170930.01	OSX10.12	DnsObsolete* & Dns_GetHostEntry*
10/2	20171002.01	Debian87	DnsObsolete* & Dns_GetHostEntry*
10/21	20171021.01	Debian87	DnsObsolete* & Dns_GetHostEntry*
10/26	20171026.01	Suse42.2	DnsObsolete* & Dns_GetHostEntry*
11/13	20171113.03	OSX10.12	DnsObsolete* & Dns_GetHostEntry*
11/16	20171116.51	SLES12	DnsObsolete* & Dns_GetHostEntry*
11/26	20171126.02	Suse42.2	DnsObsolete* & Dns_GetHostEntry*

Similar failures to dotnet/corefx#20245, but at different times (probably runs on different worker machine).

Runfo Tracking Issue: dnsobsoletegethostbyname_emptystring_returnshostname

Build	Definition	Kind	Run Name

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

[Sunny-pu]

[EDIT] Test failure included in history in top post.

[karelz]

Updated test failure history in top post.

[Sunny-pu]

failed again: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20180419.01/workItem/System.Net.NameResolution.Functional.Tests

[benaadams]

Seen in https://github.com/dotnet/corefx/pull/35267

Debian.8.Amd64.Open-x64-Release

Message :
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
Stack Trace :
   at System.Net.Dns.InternalGetHostByName(String hostName) in /__w/1/s/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 68
   at System.Net.Dns.GetHostByName(String hostName) in /__w/1/s/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 41
   at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /__w/1/s/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 108

[AriNuer]

Same test failed with similar exception:

Failed test: System.Net.NameResolution.Tests.GetHostByNameTest/DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName

Message:

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known

Stack Trace :

   at System.Net.Dns.InternalGetHostByName(String hostName) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 68
   at System.Net.Dns.ResolveCallback(Object context) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 218
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source) in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 69
   at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 361
   at System.Net.Dns.EndGetHostByName(IAsyncResult asyncResult) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 383
   at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName() in /_/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 119

Build: -20190617.81(Master) Failing configurations:

• RedHat.6.Amd64.Open-x64-Release
• Debian.9.Amd64.Open-x64-Release

Details: https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fheads~2Fmaster/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190617.81/workItem/System.Net.NameResolution.Functional.Tests/analysis/xunit/System.Net.NameResolution.Tests.GetHostByNameTest~2FDnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName

[karelz]

Triage: We should know why DNS resolution does not work sometimes -- we know it is not cached on Linux/Mac, but why does it happen so often in CI? Is it network flakiness? Or busy machine?

Plan: Add production diagnostics into DNS, then make runs with those changes to catch it. It will be useful also for production diagnostic by customers.

[wfurt]

no failure on Linux for 90 days. Some tests are still disabled on OSX and ARM64 (tracked by #27622).

[karelz]

@wfurt do we have any tests disabled against this specific bug? If not, let's close it.

[wfurt]

We have four tests disabled at this moment for OSX and the same for ARM64 (tracked in #27622).

I suspect most of the failures are not product bugs and are caused by test environment.
AFAIK we changed our macOS infrastructure as well as ARM64 support matured. How about moving the test to Outerloop? That would allow us to collect some stability data without impacting everyone's PRs. That also seems appropriate since tests do have a dependency on external service.

cc: @dotnet/ncl in case somebody has a better suggestion.

[ViktorHofer]

A bunch of NameResolution tests failed again (including the ones mentioned in this issue): https://dnceng.visualstudio.com/public/_build/results?buildId=525372&view=ms.vss-test-web.build-test-results-tab&runId=16621996&resultId=137716&paneView=debug

@wfurt @davish suggestions for mitigation?

[davidsh]

@wfurt @davish suggestions for mitigation?

Doing name resolution in a CI environment will always be flaky most likely.

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known

Perhaps we should consider an approach being done here in #32501 and only run these tests on a less regular basis. I.e.:

[Trait(XunitConstants.Category, XunitConstants.IgnoreForCI)] // DNS is flaky

FWIW, Our DNS APIs have been relatively stable from a product perspective and don't see a lot of risk running these tests less often.

[wfurt]

We should work towards stabilizing core services (like DNS) in CI.

[ViktorHofer]

What about leveraging the RetryHelper for DNS tests which are inherently flaky?

[davidsh]

We should work towards stabilizing core services (like DNS) in CI.

What about leveraging the RetryHelper for DNS tests which are inherently flaky?

Do we understand whether the problem is intermittent (i.e. retrying tests will help) or due to a mis-configuration of some kind in CI (which retrying tests will not help).

[ViktorHofer]

Do we understand whether the problem is intermittent (i.e. retrying tests will help) or due to a mis-configuration of some kind in CI (which retrying tests will not help).

I assumed it's intermittent but how would we find that out?

[davidsh]

I assumed it's intermittent but how would we find that out?

Let's add some RetryHelper logic to some of the tests. Then run it in CI. If we see a significant improvement in test reliability, then we will know that the problem is intermittent.

[Anipik]

failed again here https://dev.azure.com/dnceng/public/_build/results?buildId=1011268&view=logs&j=b9b7de4d-2126-5581-6453-1576f9aaac5f&t=db048d48-40e8-5399-9b14-d587763f33ff

[eerhardt]

Here too: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-48667-merge-af809879612c4371b8/System.Net.NameResolution.Functional.Tests/console.826d39cc.log?sv=2019-07-07&se=2021-03-17T22%3A53%3A24Z&sr=c&sp=rl&sig=qfUTGHjOCkAigQRcROLrLRTtq5MphSvppEvcO8QVJUU%3D

[ViktorHofer]

Tests were disabled on SLES with https://github.com/dotnet/runtime/pull/48759.

[VincentBu]

Failed again in runtime 20210624.123

Failed tests:

net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open

- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(hostName: \"\")

Error message
System.Exception : Failed for empty hostname.
Dns.GetHostName() == a000AID
getHostEntryException==System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
at System.Net.Dns.GetHostEntry(String hostNameOrAddress, AddressFamily family) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 91
at System.Net.Dns.GetHostEntry(String hostNameOrAddress) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 60
at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 45
etcHostsException==
/etc/host ==

hosts         This file describes a number of hostname-to-address
mappings for the TCP/IP subsystem.  It is mostly
used at boot time, when no name servers are running.
On small systems, this file can be used instead of a
"named" name server.
Syntax:
IP-Address  Full-Qualified-Hostname  Short-Hostname
127.0.0.1	localhost

special IPv6 addresses
::1             localhost ipv6-localhost ipv6-loopback

fe00::0         ipv6-localnet

ff00::0         ipv6-mcastprefix
ff02::1         ipv6-allnodes
ff02::2         ipv6-allrouters
ff02::3         ipv6-allhosts

Added by SMT registration do not remove, retain comment as well
52.148.152.22	smt-azure.susecloud.net	smt-azure

---- System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known


Stack trace
   at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 64
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
   at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
   at System.Net.Dns.GetHostEntry(String hostNameOrAddress, AddressFamily family) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 91
   at System.Net.Dns.GetHostEntry(String hostNameOrAddress) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 60
   at System.Net.NameResolution.Tests.GetHostEntryTest.<>c__DisplayClass1_0.<Dns_GetHostEntry_HostString_Ok>b__0() in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 33
   at System.Net.NameResolution.Tests.GetHostEntryTest.TestGetHostEntryAsync(Func`1 getHostEntryFunc) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 88
   at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 33

net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open

- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName

net6.0-Linux-Release-x64-Mono_release-SLES.12.Amd64.Open

- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName

Error message
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known


Stack trace
   at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
   at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
   at System.Net.Dns.GetHostByName(String hostName) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 271
   at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 109

net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open

- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_HostString_Ok(hostName: \"\")

Error message
System.AggregateException : One or more errors occurred. (One or more errors occurred. (Name or service not known)) (One or more errors occurred. (Name or service not known))
---- System.AggregateException : One or more errors occurred. (Name or service not known)
-------- System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
---- System.AggregateException : One or more errors occurred. (Name or service not known)
-------- System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known


Stack trace
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 80
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 49
   at System.Net.NameResolution.Tests.GetHostEntryTest.TestGetHostEntryAsync(Func`1 getHostEntryFunc) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 91
   at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 80
--- End of stack trace from previous location ---
----- Inner Stack Trace #1 (System.AggregateException) -----

----- Inner Stack Trace -----
   at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
   at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
   at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__33_1(Object s) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 591
   at System.Net.Dns.<>c__DisplayClass39_0`1.<RunAsync>b__0(Task <p0>, Object <p1>) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 688
   at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 94
   at System.Threading.Tasks.Task.<>c.<.cctor>b__284_0(Object obj) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2369
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 293
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2331
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 63
----- Inner Stack Trace #2 (System.AggregateException) -----

----- Inner Stack Trace -----
   at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
   at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
   at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__33_1(Object s) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 591
   at System.Net.Dns.<>c__DisplayClass39_0`1.<RunAsync>b__0(Task <p0>, Object <p1>) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 688
   at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 94
   at System.Threading.Tasks.Task.<>c.<.cctor>b__284_0(Object obj) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2369
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 293
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2331

[VincentBu]

Failed again in runtime 20210704.33

Failed tests:

net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open

- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(hostName: \"\")
- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_HostString_Ok(hostName: \"\")

One of Error messages:

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known


Stack trace
   at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
   at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
   at System.Net.Dns.GetHostByName(String hostName) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 271
   at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 109