[9.0-staging] Unable to open dump json file

[carlossanlop]

@hoyosjs Unable to access the dump json file.

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1016432 Build error leg or test failing: profiler/handles/handles/handles.sh

Error Message

{
  "ErrorMessage" : "Problem launching createdump",
  "BuildRetry" : false,
  "ExcludeConsoleLog" : false
}

• PR: https://github.com/dotnet/runtime/pull/114662
• Queue: coreclr Pri0 Runtime Tests Run linux arm64 checked
• Job result: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1016432&view=logs&j=15d7b3ca-f7a5-5373-9266-3c1554d785ae&t=d1ceec29-6ebd-5c8e-2a21-7abb8dff8831&l=37
• Log file: https://helixr18s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-release-90-staging-0950d4c86688440188/profiler.handles/1/console.27506fd3.log?helixlogtype=result
• Output:

During a test timeout, successfully collect a dump file for it:

 Starting:    profiler.handles.XUnitWrapper (parallel test collections = on [2 threads], stop on fail = off)
   profiler.handles.XUnitWrapper: [Long Running Test] 'profiler/handles/handles/handles.sh', Elapsed: 00:03:59
   profiler.handles.XUnitWrapper: [Long Running Test] 'profiler/handles/handles/handles.sh', Elapsed: 00:06:00
   profiler.handles.XUnitWrapper: [Long Running Test] 'profiler/handles/handles/handles.sh', Elapsed: 00:08:00
Attempting to collect crash dump: /home/helixbot/dotnetbuild/dumps/crashdump_48.dmp
Invoking: sudo /root/helix/work/correlation/createdump --crashreport --name "/home/helixbot/dotnetbuild/dumps/crashdump_48.dmp" 48 --withheap
   profiler.handles.XUnitWrapper: [Long Running Test] 'profiler/handles/handles/handles.sh', Elapsed: 00:10:00
createdump stdout:
[createdump] Gathering state for process 48 corerun
[createdump] Writing crash report to file /home/helixbot/dotnetbuild/dumps/crashdump_48.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/crashdump_48.dmp
[createdump] Written 97964032 bytes (23917 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 340ms

createdump stderr:

Invoking: sudo ls -l /home/helixbot/dotnetbuild/dumps/crashdump_48.dmp.crashreport.json
stdout: -rw------- 1 root root 29218 Apr 15 04:39 /home/helixbot/dotnetbuild/dumps/crashdump_48.dmp.crashreport.json

But then when we try to access the crashreport.json, access is denied:

=========================================
Error reading /home/helixbot/dotnetbuild/dumps/crashdump_48.dmp.crashreport.json: System.UnauthorizedAccessException: Access to the path '/home/helixbot/dotnetbuild/dumps/crashdump_48.dmp.crashreport.json' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
   at System.IO.File.ReadAllText(String path, Encoding encoding)
   at CoreclrTestLib.CoreclrTestWrapperLib.TryPrintStackTraceFromCrashReport(String crashReportJsonFile, TextWriter outputWriter)
Collected crash dump: /home/helixbot/dotnetbuild/dumps/crashdump_48.dmp
    profiler/handles/handles/handles.sh [FAIL]
      Problem launching createdump (may not have execute permissions): execve(���ϰ) FAILED No such file or directory (2)
      
      Assert failure(PID 57 [0x00000039], Thread: 64 [0x0040]): !PreemptiveGCDisabled()
          File: /__w/1/s/src/coreclr/vm/threads.cpp:911
          Image: /root/helix/work/correlation/corerun

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1016432 Error message validated: [[Long Running Test] 'profiler/handles/handles/handles.sh'] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 4/15/2025 3:41:30 PM UTC

Report

Build	Definition	Test	Pull Request
1016432	dotnet/runtime	profiler/handles/handles/handles.sh

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	1

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1016432 Error message validated: [Access to the path '.*crashreport.json' is denied.] Result validation: :x: Known issue did not match with the provided build. Validation performed at: 4/22/2025 8:52:38 PM UTC

Report

Build	Definition	Test	Pull Request
1016432	dotnet/runtime	profiler/handles/handles/handles.sh

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	1

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1016432 Error message validated: [Problem launching createdump] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 4/22/2025 8:55:30 PM UTC

Report

Build	Definition	Test	Pull Request
1016432	dotnet/runtime	profiler/handles/handles/handles.sh

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	1

Report

Build	Definition	Test	Pull Request
1222225	dotnet/runtime	profiler/handles/handles/handles.sh
1208805	dotnet/runtime	profiler/handles/handles/handles.sh

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	2

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @tommcdon See info in area-owners.md if you want to be subscribed.

[tommcdon]

@mdh1418

[mdh1418]

The profiler test itself passed Profilee STDOUT: PROFILER TEST PASSES, but for some reason the process isn't exiting given theres no Expected: 100\nActual:100 that is supposed to output in the test script.

This happens occasionally, and is not clear how to reproduce, I don't believe this is specific to the handles profiler test at all, but something going awry with the child process exiting.

There's some extra logging that we've added to CoreclrTestWrapperLib that didn't make it to release/9.0-staging, so for example, if the environment var for USER is not set, that might be why the crashdump couldn't be opened. @kunalspathak Do you know what's responsible for setting the USER env var?

[mdh1418]

The test timeout is not unique: https://github.com/dotnet/runtime/issues/66174 https://github.com/dotnet/runtime/issues/107067

Closing the profiler/handles/handles/handles.sh side of this issue.

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @hoyosjs See info in area-owners.md if you want to be subscribed.