diagnostics
diagnostics copied to clipboard
dotnet
Missing symbols for only dotnet code
I write minimal dotnet app and try get trace for it
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
var app = builder.Build();
app.UseSwagger();
app.UseSwaggerUI();
app.MapControllers();
app.MapGet("test_slowpoke", () =>
{
long counter = 0;
for (int i = 0; i < 1_000_000_000; i++)
{
counter++;
}
return "Hello world";
});
app.Run();
But in result trace you can wee missing symbols
record trace by command:
dotnet-trace collect --duration 00:00:00:20 -- WebApplication1
statistic:
dotnet-trace report WebApplication1_20230217_193155.nettrace topN -n 10
Top 10 Functions (Exclusive) Inclusive Exclusive
1. Missing Symbol 58.22% 58.19%
2. WaitHandle.WaitOneNoCheck(int32) 10.13% 10.13%
3. Thread.Sleep(int32) 10.09% 10.09%
4. ManualResetEventSlim.Wait(int32,value class System.Threading.Cancellat 10.08% 10.08%
5. webapplication1!Program+<>c.<<Main>$>b__0_0() 1.13% 1.13%
6. WebHost.ConfigureWebDefaults(class Microsoft.AspNetCore.Hosting.IWebHo 0.02% 0.01%
7. .cctor() 0.01% 0.01%
8. corelib!Interop+Sys.LStat(unsigned int8&,value class FileStatus&) 0% 0%
9. Http1ParsingHandler].TryTakeSingleHeader(!0,value class System.ReadOnl 0.01% 0%
10. RuntimeType.CreateInstanceOfT() 0.01% 0%
Tell me what symbols you need to load to see the statistics
my machine PRETTY_NAME="Ubuntu 22.04.1 LTS" architecture x86 dotnet 7
Tagging subscribers to this area: @tommcdon See info in area-owners.md if you want to be subscribed.
Issue Details
I write minimal dotnet app and try get trace for it
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
var app = builder.Build();
app.UseSwagger();
app.UseSwaggerUI();
app.MapControllers();
app.MapGet("test_slowpoke", () =>
{
long counter = 0;
for (int i = 0; i < 1_000_000_000; i++)
{
counter++;
}
return "Hello world";
});
app.Run();
But in result trace you can wee missing symbols
record trace by command:
dotnet-trace collect --duration 00:00:00:20 -- WebApplication1
statistic:
dotnet-trace report WebApplication1_20230217_193155.nettrace topN -n 10
Top 10 Functions (Exclusive) Inclusive Exclusive
1. Missing Symbol 58.22% 58.19%
2. WaitHandle.WaitOneNoCheck(int32) 10.13% 10.13%
3. Thread.Sleep(int32) 10.09% 10.09%
4. ManualResetEventSlim.Wait(int32,value class System.Threading.Cancellat 10.08% 10.08%
5. webapplication1!Program+<>c.<<Main>$>b__0_0() 1.13% 1.13%
6. WebHost.ConfigureWebDefaults(class Microsoft.AspNetCore.Hosting.IWebHo 0.02% 0.01%
7. .cctor() 0.01% 0.01%
8. corelib!Interop+Sys.LStat(unsigned int8&,value class FileStatus&) 0% 0%
9. Http1ParsingHandler].TryTakeSingleHeader(!0,value class System.ReadOnl 0.01% 0%
10. RuntimeType.CreateInstanceOfT() 0.01% 0%
Tell me what symbols you need to load to see the statistics
my machine PRETTY_NAME="Ubuntu 22.04.1 LTS" architecture x86 dotnet 7
| Author: | careless6666 |
|---|---|
| Assignees: | - |
| Labels: |
|
| Milestone: | - |
Hi @careless6666,
I just tried to run your program and could not reproduce the missing symbol issue, although that is probably because I have symbols for dotnet set up.
I get this output:
Top 10 Functions (Exclusive) Inclusive Exclusive
1. il!Interop+Kernel32.GetQueuedCompletionStatus(int,unsigned int32&,unsi 35.75% 35.75%
2. PortableThreadPool+IOCompletionPoller.Poll() 21.05% 21.05%
3. WaitHandle.WaitOneNoCheck(int32) 10.46% 10.46%
4. Thread.Sleep(int32) 10.42% 10.42%
5. ManualResetEventSlim.Wait(int32,value class System.Threading.Cancellat 10.41% 10.41%
6. test_webapp!Program+<>c.<<Main>$>b__0_0() 1.15% 1.15%
7. __Canon].get_Values() 0.06% 0.06%
8. test_webapp!Program.<Main>$(class System.String[]) 10.55% 0.03%
9. String.Ctor(wchar*) 0.02% 0.02%
10. RuntimeTypeHandle.GetTypeByName(class System.String,bool,bool,value cl 0.01% 0.01%
So it seems like you are missing symbols for il!Interop and PortableThreadPool, which are both in the dotnet runtime.
The topN command current does not allow you to set the symbol path directly, can you try setting _NT_SYMBOL_PATH=srv*c:\DownstreamStore*https://msdl.microsoft.com/download/symbols as described here?
Sorry, but you give me a link to the documentation for Windows. I have dotnet-symbol installed on my linux, I try to download all symbols for my trace by
dotnet-symbol WebApplication1_20230217_192413.nettrace --microsoft-symbol-server
When i check my cache symbol cache folder and nothing changed
~/.dotnet/symbolcache$ ls -la
total 32
drwxrwxr-x 8 alex alex 4096 jan 18 15:39 .
drwxrwxr-x 10 alex alex 4096 feb 27 09:37 ..
drwxrwxr-x 3 alex alex 4096 jan 18 15:38 libclrjit.so
drwxrwxr-x 4 alex alex 4096 jan 18 15:39 libmscordaccore.so
drwxrwxr-x 3 alex alex 4096 jan 18 15:39 microsoft.extensions.hosting.abstractions.dll
drwxrwxr-x 3 alex alex 4096 jan 18 15:39 microsoft.extensions.hosting.abstractions.pdb
drwxrwxr-x 3 alex alex 4096 jan 18 15:39 system.private.corelib.dll
drwxrwxr-x 4 alex alex 4096 jan 18 15:39 system.private.corelib.pdb
I setup _NT_SYMBOL_PATH
export _NT_SYMBOL_PATH=/home/alex/.dotnet/symbolcache
echo $_NT_SYMBOL_PATH
/home/alex/.dotnet/symbolcache
But it did not help
I tried downloading symbols manually from the site(https://msdl.microsoft.com/download/symbols), but it seems that this method is currently not available from the browser
I don't believe dotnet-symbol supports trace files as an argument, you would need to either pass in a dump file of your app or the dotnet binaries (libcoreclr.so, System.Private.Corelib.dll, etc) that would be at $DOTNET_ROOT/shared/Microsoft.NETCore.App/<version>/
I guess the problem is still in me, although I don't know what I'm doing wrong
I download all symbols
for i in `ls` ; do echo `dotnet-symbol $i -o /home/alex/.dotnet/symbolcache` ; done
ls /home/alex/.dotnet/symbolcache
createdump.dbg
System.IO.FileSystem.Primitives.pdb
System.Resources.ResourceManager.pdb
system.io.filesystem.watcher.ni.r2rmap
system.resources.writer.ni.r2rmap
libclrgc.so.dbg
System.IO.FileSystem.Watcher.ni.r2rmap
System.Resources.Writer.ni.r2rmap
libclrjit.so
system.io.filesystem.watcher.pdb
system.resources.writer.pdb
libclrjit.so.dbg
System.IO.FileSystem.Watcher.pdb
System.Resources.Writer.pdb
libcoreclr.so.dbg
system.io.isolatedstorage.ni.r2rmap
system.runtime.compilerservices.unsafe.pdb
libcoreclrtraceptprovider.so.dbg
System.IO.IsolatedStorage.ni.r2rmap
System.Runtime.CompilerServices.Unsafe.pdb
libhostpolicy.so.dbg
system.io.isolatedstorage.pdb
system.runtime.compilerservices.visualc.ni.r2rmap
libmscordaccore.so
System.IO.IsolatedStorage.pdb
System.Runtime.CompilerServices.VisualC.ni.r2rmap
libmscordaccore.so.dbg
system.io.memorymappedfiles.ni.r2rmap
system.runtime.compilerservices.visualc.pdb
libmscordbi.so.dbg
System.IO.MemoryMappedFiles.ni.r2rmap
System.Runtime.CompilerServices.VisualC.pdb
libSystem.Globalization.Native.so.dbg
system.io.memorymappedfiles.pdb
system.runtime.extensions.pdb
libSystem.IO.Compression.Native.so.dbg
System.IO.MemoryMappedFiles.pdb
System.Runtime.Extensions.pdb
libSystem.Native.so.dbg
system.io.pdb
system.runtime.handles.pdb
libSystem.Net.Security.Native.so.dbg
System.IO.pdb
System.Runtime.Handles.pdb
libSystem.Security.Cryptography.Native.OpenSsl.so.dbg
system.io.pipes.accesscontrol.ni.r2rmap
system.runtime.interopservices.javascript.ni.r2rmap
microsoft.csharp.ni.r2rmap
System.IO.Pipes.AccessControl.ni.r2rmap
System.Runtime.InteropServices.JavaScript.ni.r2rmap
Microsoft.CSharp.ni.r2rmap
system.io.pipes.accesscontrol.pdb
... long list
It seems I downloaded all possible pdb but the result is the same, missing symbols
It may not be your issue, we didn't provide an option to specify the symbol path as part of dotnet-trace report topN, which was an oversight on our part. I was trying to see if there was a way we could work around it with _NT_SYMBOL_PATH but it looks like we can't.
Would you be willing to try a private build of dotnet-trace with changes? This version of dotnet-trace has the additional argument --additional-symbol-paths that you can provide a comma separated list of paths to look for symbols in. https://github.com/davmason/diagnostics/releases/tag/v1.0.0-private-dotnet-trace
You should be able to download, unzip and run it on an existing trace
I tried everything as in that branch, checking that the path is the one that should be, but the result is the same, can you attach your symbols for comparison, what is missing? My symbols https://drive.google.com/file/d/1bOHE342bHJCYLV53FadT1BphPKInswhF/view?usp=sharing value from symbol paths variable:
;SRV*https://msdl.microsoft.com/download/symbols;SRV*https://nuget.smbsrc.net;SRV*https://referencesource.microsoft.com/symbols;/home/alex/.dotnet/symbolcache
and receive same result
Can you try with the private dotnet-trace and without the srv*? Like dotnet-trace report <trace> topN --additionalsymbolpaths https://nuget.smbsrc.net;https://referencesource.microsoft.com/symbols;/home/alex/.dotnet/symbolcache
I took the code from your branch and replaced the default source with https://nuget.smbsrc.net;https://referencesource.microsoft.com/symbols passed his characters through the parameter
Maybe it would be better if I wrap my example in docker and write here code of docker file?
@careless6666 did you try using the --additionalsymbolpaths argument and passing in your /home/alex/.dotnet/symbolcache path?
I'm out of ideas then, I would expect it to work with all the symbols downloaded using dotnet-symbol.
Hi @careless6666! We are not able to reproduce the issue locally. Would you mind sharing a trace + the related symbols so that we can reproduce the issue? Thanks in advance!
This issue has been marked needs-author-action and may be missing some important information.
I hope i put everything that need in this repository https://github.com/careless6666/symbols
Thanks @careless6666 - I am able to reproduce the issue opening the nettrace file. This looks like it is due to a bug in dotnet-trace.
The dotnet-trace "TopN" feature is effectively the same as opening the nettrace in perfview, opening the "Thread Time" view, then removing any Group Patterns, Fold%, and replacing FoldPaths with CPU_TIME;UNMANAGED_CODE_TIME;{Thread (}.
Opeing the trace:
Viewing the "By Name" view with the above settings:
Note that the ?!? function is listed at the top with the most Exclusive samples and represents the Missing Symbol item in dotnet-trace (see https://github.com/dotnet/diagnostics/blob/95221a2f90977f8fb93ab58b2f252c0bb0c84e8b/src/Tools/dotnet-trace/CommandLine/PrintReportHelper.cs#L122-L125).
Viewing the Call Tree view we can see that ?!? is the leaf node of several callstacks, for example:
Removing UNMANAGED_CODE_TIME from FoldPaths uncovers where the time is actually being spent in managed code, for example:
The ?!? function is likely a PInvoke stub created by the runtime that enables it to call from managed to unmanaged code.
So the root issue here is that dotnet-trace is 1) displaying thread time and not CPU time (tracked on https://github.com/dotnet/diagnostics/issues/3060 and https://github.com/dotnet/runtime/issues/68446), 2) incorrectly attributing time in unmanaged code to the PInvoke calling stub.
To address this particular issue, we could try symbolizing the stub (perhaps calling it Unmanaged Stub) in traceevent (part of https://github.com/microsoft/perfview) and then folding that symbol in dotnet-trace. Since the fixes are likely in tracevent/dotnet-trace, I'll move this issue over to the https://github.com/dotnet/diagnostics repo. If any runtime or traceevent work is needed we can create issue for those specific issues.