Support for printing thread stack traces and heaps composition in dotnet-monitor

[mleenhardt]

We wrote a much simpler version of dotnet monitor that uses Microsoft.Diagnostics.Runtime to help with high level debugging of remote processes running in Kubernetes. Our implementation returns a stack trace of all threads in a given process, as well as the type composition of all heaps, ordered by object size. This is very convenient to get a feel for what a process is doing without having to download a dump and load it in dotnet analyze.

e.g. In our implementation, http://localhost:52323/dump returns something that looks like

--------------------------------------------------------------------
    STACK TRACE
--------------------------------------------------------------------
Thread - OS 00015910 - Managed 1:
  25aaf7e070 7ffb6af86278 [GCFrame]
  25aaf7e1b8 7ffb6af86278 [HelperMethodFrame_1OBJ]
  25aaf7e2e0 7ffbc9a43725 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
  25aaf7e380 7ffbc9a43052 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
  25aaf7e3f0 7ffbc9a42dcf System.Threading.Tasks.Task.InternalWaitCore(Int32, System.Threading.CancellationToken)
  25aaf7e470 7ffbc9a61604 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
  25aaf7e4a0 7ffbc9a4432d System.Runtime.CompilerServices.TaskAwaiter.GetResult()
  25aaf7e4d0 7ffc275d6585 Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(Microsoft.Extensions.Hosting.IHost)
  25aaf7e500 7ffb6a5a1790 WebApplication2.Program.Main(System.String[])
  25aaf7e788 7ffbca0c6103 [GCFrame]
  25aaf7ed30 7ffbca0c6103 [GCFrame]

Thread - OS 00002a90 - Managed 2:
  25abbffa00 7ffb6af86278 [DebuggerU2MCatchHandlerFrame]

Thread - OS 00008898 - Managed 4:
  25ac37ed18 7ffb6af86278 [HelperMethodFrame]
  25ac37ee10 7ffbc9b2b91b System.Threading.Thread.Sleep(Int32)
  25ac37ee40 7ffb6a5a4535 WebApplication2.Program.DoWork()
  25ac37eeb0 7ffb6a5a40d1 WebApplication2.Program+<>c.<Main>b__0_0()
  25ac37eee0 7ffbc9a47bd3 System.Threading.Tasks.Task.InnerInvoke()
  25ac37ef20 7ffbc9a47b82 System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object)
  25ac37ef50 7ffbc9a47b02 System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
  25ac37efa0 7ffbc9a477d7 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
  25ac37f040 7ffbc9a476b3 System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
  25ac37f080 7ffbc9a4764b System.Threading.Tasks.Task.ExecuteFromThreadPool(System.Threading.Thread)
  25ac37f0b0 7ffbc9a4723c System.Threading.ThreadPoolWorkQueue.Dispatch()
  25ac37f160 7ffbc9a4706b System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
  25ac37f570 7ffbca0c6103 [DebuggerU2MCatchHandlerFrame]

[...]

--------------------------------------------------------------------
    HEAPS
--------------------------------------------------------------------

--------------------------------------------------------------------
GENERATION 0
System.String - 764954
System.Byte[] - 735445
System.Security.Cryptography.Oid - 735360
System.Char[] - 419616
System.Security.Cryptography.X509Certificates.X509Extension[] - 201128
System.Security.Cryptography.X509Certificates.X509Extension - 193440
[...]

--------------------------------------------------------------------
GENERATION 1
Free - 288

--------------------------------------------------------------------
GENERATION 2
Free - 768

Would it be feasible or would you be interested in supporting a similar feature? Maybe in two different endpoints /stacktrace and /heap?

[rwkarg]

A diagnostics summary like this would be nice to have available without needing to work with a dump or worry about the target architecture.

[rwkarg]

For an example use cases that would be interesting to me:

We will occasionally have (third-party, of course) dependencies that have un-synchronized access to a non-thread safe collection (like a Dictionary<K, V>). If we see three cores pegged on the process and thread stacks show three threads with a Dictionary<,>.Insert stack trace then we can easily diagnose where that is happening and push a fix to the upstream.

[shirhatti]

cc @josalem who was writing a similar tool a while ago- https://github.com/josalem/DotStack

[davidfowl]

I like this feature but I'd like to return a JSON representation of the stacks not a textual one so that we could build a nice UI on top (someday) 😄

[rwkarg]

I like this feature but I'd like to return a JSON representation of the stacks not a textual one so that we could build a nice UI on top (someday) 😄

Accept ftw

[sukesh-ak]

I could not get the sample by @josalem working, I guess due to change in API with unsupported format error etc. So I put together a sample with EventPipe trace code and stack related code borrowed from his sample. https://github.com/sukesh-ak/EventPipe-Diagnostics

Do let me know if you have any suggestions.

[jander-msft]

FYI, we've added a /stacks route to dotnet monitor in 7.0 RC 1 as an experimental feature:

• Release: https://github.com/dotnet/dotnet-monitor/releases/tag/v7.0.0-rc.1.22479.8
• Docs: https://github.com/dotnet/dotnet-monitor/blob/main/documentation/api/stacks.md

At the release of 7.0, it will remain an experimental feature.

We are continuing to invest in this feature and hope to fully support it in a post-7.0 release. Please give it a try and let us know about any feedback you have for it!