runtime icon indicating copy to clipboard operation
runtime copied to clipboard

tracing/eventpipe/providervalidation/providervalidation/providervalidation.cmd test failure

Open ViktorHofer opened this issue 1 year ago • 1 comments

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=661154 Build error leg or test failing: tracing/eventpipe/providervalidation/providervalidation/providervalidation.cmd Pull request: https://github.com/dotnet/runtime/pull/101715

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "./nptl/futex-internal.c: No such file or directory",
  "ErrorPattern": "",
  "BuildRetry": true,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=661154 Error message validated: [./nptl/futex-internal.c: No such file or directory] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 5/1/2024 8:59:40 AM UTC

Report

Build Definition Test Pull Request
724846 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
724500 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
724452 dotnet/runtime System.Numerics.Vectors.Tests.WorkItemExecution dotnet/runtime#104049
724271 dotnet/runtime System.Text.Json.SourceGeneration.Roslyn4.4.Tests.WorkItemExecution dotnet/runtime#104069
723841 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution dotnet/runtime#104104
723821 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
723303 dotnet/runtime System.Text.RegularExpressions.Tests.WorkItemExecution dotnet/runtime#103757
723092 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
723260 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
722988 dotnet/runtime JSImportGenerator.Unit.Tests.WorkItemExecution dotnet/runtime#104115
722226 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
721655 dotnet/runtime System.Numerics.Vectors.Tests.WorkItemExecution dotnet/runtime#104049
721660 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
720992 dotnet/runtime ComInterfaceGenerator.Unit.Tests.WorkItemExecution dotnet/runtime#103412
720779 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
720689 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution dotnet/runtime#104009
720314 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
720283 dotnet/runtime ComInterfaceGenerator.Unit.Tests.WorkItemExecution dotnet/runtime#103757
719372 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
719184 dotnet/runtime System.Runtime.CompilerServices.Unsafe.Tests.WorkItemExecution dotnet/runtime#103915
718997 dotnet/runtime System.Runtime.CompilerServices.Unsafe.Tests.WorkItemExecution dotnet/runtime#103915
719043 dotnet/runtime System.Text.Json.Tests.WorkItemExecution dotnet/runtime#103757
718755 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
718708 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution dotnet/runtime#103372
717715 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
717491 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
717200 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
717011 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
716673 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
716357 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
716194 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
715648 dotnet/runtime Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution dotnet/runtime#103635
715312 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
715268 dotnet/runtime System.Text.Json.Tests.WorkItemExecution dotnet/runtime#103617
714740 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
714490 dotnet/runtime Microsoft.Extensions.Logging.EventSource.Tests.WorkItemExecution dotnet/runtime#103655
713808 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
713359 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
711960 dotnet/runtime Microsoft.Extensions.Logging.EventSource.Tests.WorkItemExecution dotnet/runtime#103655
712515 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
712474 dotnet/runtime System.Private.Uri.Functional.Tests.WorkItemExecution dotnet/runtime#102144
712361 dotnet/runtime System.Text.RegularExpressions.Tests.WorkItemExecution dotnet/runtime#102655
711823 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution dotnet/runtime#103502
711514 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution dotnet/runtime#103635
711308 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
710961 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution dotnet/runtime#103608
710799 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
710377 dotnet/runtime Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution dotnet/runtime#103565
709764 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
709579 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
709287 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
709132 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
708778 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
708366 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
707642 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
707186 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution dotnet/runtime#103308
707135 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
707003 dotnet/runtime LibraryImportGenerator.Unit.Tests.WorkItemExecution dotnet/runtime#103436
706379 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
706387 dotnet/runtime System.Text.Json.SourceGeneration.Roslyn4.4.Tests.WorkItemExecution dotnet/runtime#103400
705872 dotnet/runtime LibraryImportGenerator.Unit.Tests.WorkItemExecution dotnet/runtime#103286
705724 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
705344 dotnet/runtime System.Text.Json.SourceGeneration.Roslyn4.4.Tests.WorkItemExecution dotnet/runtime#102882
704840 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
704223 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
704278 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution dotnet/runtime#99747
703418 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
703199 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution dotnet/runtime#102364
702950 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
702688 dotnet/runtime System.Text.Json.SourceGeneration.Roslyn4.4.Tests.WorkItemExecution dotnet/runtime#103228
702087 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
701895 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
701896 dotnet/runtime Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution dotnet/runtime#103212
701662 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
701555 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
701345 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
700982 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
700825 dotnet/runtime System.Text.Json.SourceGeneration.Roslyn4.4.Tests.WorkItemExecution dotnet/runtime#103159
700465 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
699938 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
699890 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
699686 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
699228 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
698629 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
697990 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
697448 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
697335 dotnet/runtime System.Runtime.Serialization.BinaryFormat.Tests.WorkItemExecution dotnet/runtime#102379
697260 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
696752 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
696224 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
695474 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
695301 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
695098 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
695011 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
694770 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
694398 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
694180 dotnet/runtime JSImportGenerator.Unit.Tests.WorkItemExecution dotnet/runtime#102795
693818 dotnet/runtime IntrinsicsInSystemPrivateCoreLib.Tests.WorkItemExecution dotnet/runtime#102917
693619 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
693568 dotnet/runtime Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution dotnet/runtime#102874
Displaying 100 of 102 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
3 28 102

ViktorHofer avatar May 01 '24 08:05 ViktorHofer

Test output from failing test in case it is useful

Raw output file:      /datadisks/disk1/work/9EDF091C/w/A6940942/uploads/eventpipe/providervalidation/providervalidation/output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/9EDF091C/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true providervalidation.dll ''
  0.0s: ==TEST STARTING==
  0.2s: Connecting to EventPipe...
  0.2s: Started sending sentinel events...
  0.3s: Creating EventPipeEventSource...
  0.3s: EventPipeEventSource created
  0.3s: Dynamic.All callback registered
  0.3s: Starting stream processing...
  0.4s: Saw new provider 'Microsoft-DotNETCore-SampleProfiler'
  0.4s: Saw sentinel event
  0.4s: Stopped sending sentinel events
  0.4s: Starting event generating action...
  0.4s: Fired MyEvent 0/100,000 times...
  0.5s: Fired MyEvent 10,000/100,000 times...
  0.5s: Saw new provider 'MyEventSource'
  0.5s: Fired MyEvent 20,000/100,000 times...
  0.6s: Fired MyEvent 30,000/100,000 times...
  0.6s: Fired MyEvent 40,000/100,000 times...
  0.7s: Fired MyEvent 50,000/100,000 times...
  0.7s: Fired MyEvent 60,000/100,000 times...
  0.8s: Fired MyEvent 70,000/100,000 times...
  0.8s: Fired MyEvent 80,000/100,000 times...
  0.9s: Fired MyEvent 90,000/100,000 times...
  0.9s: Stopping event generating action
  0.9s: Sending StopTracing command...
  1.0s: Saw new provider 'Microsoft-DotNETCore-EventPipe'
  1.0s: Saw new provider 'Microsoft-Windows-DotNETRuntimeRundown'

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x76e90ccf248f - Unknown
	0x76e90cc9447e - Unknown
	0x76e90cbf87f1 - Unknown
	0x76e90dc42520 - Unknown
	0x41f904f6 - Unknown
  1.1s: Finished StopTracing command

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x41f904f6):0x41f904e6  f9 ff 49 8b c6 49 83 c6 04 49 8b cd 49 83 c5 04  ..I..I...I..I...
0x41f904f6  48 63 09 89 08 41 ff cf 45 85 ff 7f cd 4c 8b 2c  Hc...A..E....L.,
0x41f90506  24 4c 8b 74 24 08 4c 8b 7c 24 10 48 83 c4 18 c3  $L.t$.L.|$.H....
0x41f90516  00 00 00 00 00 00 00 00 00 00 48 83 ec 28 4c 89  ..........H..(L.

=================================================================
	Managed Stacktrace:
=================================================================
	  at Microsoft.Diagnostics.Tracing.TraceEvent:CopyBlob <0x00066>
	  at Microsoft.Diagnostics.Tracing.TraceEvent:Clone <0x002e3>
	  at Microsoft.Diagnostics.Tracing.Parsers.ExternalTraceEventParser:<.ctor>b__0_0 <0x000a1>
	  at Microsoft.Diagnostics.Tracing.TraceEventDispatcher:Lookup <0x00355>
	  at Microsoft.Diagnostics.Tracing.EventPipeEventSource:DispatchEventRecord <0x00047>
	  at Microsoft.Diagnostics.Tracing.EventPipeEventSource:EventCache_OnEvent <0x000ef>
	  at Microsoft.Diagnostics.Tracing.EventPipe.EventCache:SortAndDispatch <0x0030a>
	  at Microsoft.Diagnostics.Tracing.EventPipe.EventCache:ProcessSequencePointBlock <0x000bf>
	  at Microsoft.Diagnostics.Tracing.EventPipeSequencePointBlock:ReadBlockContents <0x00083>
	  at Microsoft.Diagnostics.Tracing.EventPipeBlock:FromStream <0x00122>
	  at FastSerialization.Deserializer:ReadObjectDefinition <0x00303>
	  at FastSerialization.Deserializer:ReadObject <0x002df>
	  at Microsoft.Diagnostics.Tracing.EventPipeEventSource:Process <0x000cb>
	  at <>c__DisplayClass9_0:<Validate>b__1 <0x00444>
	  at System.Threading.Tasks.Task:InnerInvoke <0x0006d>
	  at <>c:<.cctor>b__292_0 <0x0002f>
	  at System.Threading.ExecutionContext:RunFromThreadPoolDispatchLoop <0x00086>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x002d7>
	  at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x000a3>
	  at System.Threading.Tasks.Task:ExecuteFromThreadPool <0x00033>
	  at System.Threading.ThreadPoolWorkQueue:Dispatch <0x003ae>
	  at WorkerThread:WorkerThreadStart <0x001bb>
	  at System.Threading.Thread:StartCallback <0x000f0>
	  at System.Object:runtime_invoke_void__this__ <0x00091>
=================================================================

tommcdon avatar May 07 '24 20:05 tommcdon

The error message regex pattern ./nptl/futex-internal.c: No such file or directory is matching failed builds on CI not related to the providervalidation eventpipe tests.

I ran providervalidation locally, and wasn't able to immediately repro the error Tom provided.

mdh1418 avatar Jul 11 '24 20:07 mdh1418

From looking at the past 9 build console logs, it looks like the futex-internal.c: No such file or directory is a product of something very similar to what @lambdageek mentioned in https://github.com/dotnet/runtime/issues/87444#issuecomment-1768952563.

All of the 9 logs' thread 1 stacks call ./nptl/pthread_cond_wait.c:627 stemming from src/native/libs/System.Native/pal_threading.c:152/src/mono/mono/mini/interp/interp.c:2368.

So it seems mono specific. Is it possible to highlight the actual reason for the crash from the External Debugger Dump section for mono instead of the misleading futex-internal.c: No such file or directory messages @thaystg @lambdageek?

@carlossanlop did you create a different tracking issue for https://github.com/dotnet/runtime/issues/87444, or should we just create error message matching at the Managed stacktrace level?

mdh1418 avatar Jul 11 '24 21:07 mdh1418

Closing for the same reason as https://github.com/dotnet/runtime/issues/87444, futex-internal.c: No such file or directory is a generic error message.

Instead, issues for CI failures should try to match part of the managed stack trace instead if the Native stacktrace is unknown and the External Debugger dump points towards "futex-internal"

mdh1418 avatar Jul 12 '24 16:07 mdh1418