Mono crashes test runs

[karelz]

8/7 - occurrences on Android, tvOS, MacCatalyst, iOS, iOSSimulator, tvOSSimulator - only those without Test Results associated

Test suite	#	Details
JIT.Directed Work Item	50x	(all without Test Results) in last 120 days - first on 8/2 JIT/Directed/aliasing_retbuf/aliasing_retbuf/aliasing_retbuf.sh test failed in logs - sometimes recognized as specific failure - see Runfo
System.Text.Json.Tests Work Item	49x	(without Test Results) in last 120 days - first one on 7/24
System.Runtime.Tests Work Item	15x	(without Test Results) in last 120 days - first on 7/27
System.Security.Cryptography.Tests Work Item	44x	(all without Test Results) in last 120 days - first on 7/24
System.ComponentModel.Tests Work Item	10x	(all without Test Results) in last 120 days - first on 8/1
System.IO.Hashing.Tests Work Item	11x	(all without Test Results) (excluded 1 with different error) in last 120 days - first on 8/1
System.IO.FileSystem.Primitives.Tests Work Item	10x	(all without Test Results) in last 120 days - first on 8/1
System.Xml.RW.WrappedReader.Tests Work Item	4x	(all without Test Results) (excluded 1 with different error) in last 120 days - first on 8/3
System.Threading.Tasks.Dataflow.Tests Work Item	22x	(without Test Results and Core Dump) in last 120 days - first on 7/26
System.Net.WebSockets.Tests Work Item	9x	(without Test Results) in last 120 days - first on 8/1
System.Linq.Parallel.Tests Work Item	~20x	(some logs have different failure) in last 120 days - first on 7/26
Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests Work Item	8x	(some logs have different failure) in last 120 days - first on 8/1
System.Runtime.Numerics.Tests Work Item	12x	(without Test Runs and Core Dump) in last 120 days - first on 7/22
System.Xml.Linq.Events.Tests Work Item	3x	(some logs have different failure) in last 120 days - first on 8/3
System.Numerics.Tensors.Tests Work Item	9x	(some logs have different failure) in last 120 days - first on 8/1
System.Net.Http.Json.Unit.Tests Work Item	11x	(some logs have different failure) in last 120 days - first on 8/1

Example:

• Rolling run on 8/7 AM - 1928273
  • JIT.Directed - Console log 1 and Console log 2
  • System.Security.Cryptography.Tests - Console log
• Rolling run on 8/16 PM - 1927956
  • JIT.Directed - Console log 1 and Console log 2
  • System.Console.Manual.Tests - Console log
  • System.Runtime.Tests - Console log

[dotnet-issue-labeler[bot]]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[karelz]

@lewing can you please help route it? This is massive test failure, causing significant noise in Rolling runs and PRs.

[msftbot[bot]]

Tagging subscribers to this area: @directhex See info in area-owners.md if you want to be subscribed.

Issue Details

---

8/7 - occurrences on Android, tvOS, MacCatalyst, iOS, iOSSimulator, tvOSSimulator - only those without Test Results associated

Test suite	#	Details
JIT.Directed Work Item	50x	(all without Test Results) in last 120 days - first on 8/2 JIT/Directed/aliasing_retbuf/aliasing_retbuf/aliasing_retbuf.sh test failed in logs - sometimes recognized as specific failure - see Runfo
System.Text.Json.Tests Work Item	49x	(without Test Results) in last 120 days - first one on 7/24
System.Runtime.Tests Work Item	15x	(without Test Results) in last 120 days - first on 7/27
System.Security.Cryptography.Tests Work Item	44x	(all without Test Results) in last 120 days - first on 7/24
System.ComponentModel.Tests Work Item	10x	(all without Test Results) in last 120 days - first on 8/1
System.IO.Hashing.Tests Work Item	11x	(all without Test Results) (excluded 1 with different error) in last 120 days - first on 8/1
System.IO.FileSystem.Primitives.Tests Work Item	10x	(all without Test Results) in last 120 days - first on 8/1
System.Xml.RW.WrappedReader.Tests Work Item	4x	(all without Test Results) (excluded 1 with different error) in last 120 days - first on 8/3
System.Threading.Tasks.Dataflow.Tests Work Item	22x	(without Test Results and Core Dump) in last 120 days - first on 7/26
System.Net.WebSockets.Tests Work Item	9x	(without Test Results) in last 120 days - first on 8/1
System.Linq.Parallel.Tests Work Item	~20x	(some logs have different failure) in last 120 days - first on 7/26
Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests Work Item	8x	(some logs have different failure) in last 120 days - first on 8/1
System.Runtime.Numerics.Tests Work Item	12x	(without Test Runs and Core Dump) in last 120 days - first on 7/22
System.Xml.Linq.Events.Tests Work Item	3x	(some logs have different failure) in last 120 days - first on 8/3
System.Numerics.Tensors.Tests Work Item	9x	(some logs have different failure) in last 120 days - first on 8/1
System.Net.Http.Json.Unit.Tests Work Item	11x	(some logs have different failure) in last 120 days - first on 8/1

Example:

• Rolling run on 8/7 AM - 1928273
  • JIT.Directed - Console log 1 and Console log 2
  • System.Security.Cryptography.Tests - Console log
• Rolling run on 8/16 PM - 1927956
  • JIT.Directed - Console log 1 and Console log 2
  • System.Console.Manual.Tests - Console log
  • System.Runtime.Tests - Console log

Author:	karelz
Assignees:	lewing
Labels:	blocking-clean-ci, untriaged, area-Infrastructure-mono, runtime-mono
Milestone:	-

[karelz]

From offline discussion with @steveisok - this is being already looked at by somebody.

[karelz]

The JIT.Directed failures are consistent in EVERY Rolling run starting with 8/2 PM 1918913 -- the error is somewhat different, is it perhaps different problem?

[akoeplinger]

@simonrozsival will look at the JIT.Directed failure as I agree it looks unrelated to the TCP tunnel problem.

[simonrozsival]

The JIT.Directed logs contain stack traces showing null pointer dereferencing when unhandled_exception_handler is called. This reminds me of https://github.com/dotnet/runtime/issues/69601.

But more importantly, the test tries to load a dynamic library and it can't find the file:

      08-06 22:57:20.830 15268 15286 E mono    : 
      08-06 22:57:20.830 15268 15286 E mono    : Unhandled Exception:
      08-06 22:57:20.830 15268 15286 E mono    : System.DllNotFoundException: AliasingRetBufNative
      08-06 22:57:20.830 15268 15286 E mono    :    at AliasingRetBuf.CallPtrPInvoke(Foo* fi)
      08-06 22:57:20.830 15268 15286 E mono    :    at AliasingRetBuf.Main()

The AliasingRetBufNative test was added recently: https://github.com/dotnet/runtime/pull/73059

[karelz]

JIT.Directed test disabled in PR #73704