symbols / pdb files.

[rwebber]

I have superdump up and running. Thank-you, this looks like just what I need. Unfortunately, I am unsure how to set the symbol path to include my pbd files. Do I need to edit a Dockerfile and rebuild? Any help here is most appreciated.

[discostu105]

You should be able to just pass in the _NT_SYMBOL_PATH environment variable in the docker run command.

I believe this should work (untested): docker run -d -p 80:5000 -e "_NT_SYMBOL_PATH=\\mySymbolServer" -v superdump:C:\superdump\data\dumps discostu105/superdump

Let me know if this works for you!

[rwebber]

Does this work for symbol files that reside on my local windows system? I expect superdump is referencing symbols internal to the container. Do externals get copied in ? How does this work? (sorry a little new to Docker) Any guidance if greatly appreciated.

[discostu105]

It does not work with symbol files that reside on your local disk. Because from within the container, there is no general access do your host-system disk, except for folders that are mounted in.

Things you can do:

• In case you have a symbol store on your host system, you could re-use that like that:
  • assuming it is "C:\symbols", you could mount the directory -v C:\symbols:C:\symbols (the same directory will be available inside the container under the same path)
  • then, use an _NT_SYMBOL_PATH that re-uses this local symbol cache. e.g. _NT_SYMBOL_PATH=srv*c:\symbols*https://msdl.microsoft.com/download/symbols
• Other than that: a secret feature of SuperDump is: if you upload a zip, which contains .pdb files, they are automatically added to the local symbol cache (inside the container!). Dumps are analyzed after that. So, you could upload zip files, containing dumps and matching pdbs and it should work fine.

[andresod]

I'm quite excited about superdump, I have been experimenting with it during the last couple of days. I had to familiarize myself first with windbg as I was running into a lot of problems that were not related to superdump. However I have now reached a point where I am stuck because I don't know docker/superdump that well.

I have a minidump that works just fine when using windbg on my local machine. If I use the same minidump in superdump, it can't find any of the symbols, except for the windows ones. However, if I first populate the symbol cache folder (which I bind via the -v option) with my local windbg then superdump (sometimes) works. Sometimes, because a lot of times, superdump will crash with outofmemory, or outofstack or some other error message. Is there a limit on the number of DLLs/PDBs that can be loaded? The 'secret feature' for uploading a zip with the .pdb files does not seem to work for me, or perhaps I am not using the correct folder structure.

[discostu105]

@andresod great that you like the project :)

Which deployment do you use?

1. docker-image
2. downloaded release binaries
3. self-built?

Can you share your symbol path and how you did set it? How did you start SuperDumpService?

In cases 2. and 3., I'd recommend to omit SuperDumpService at first. Use build/bin/SuperDumpx64/SuperDump.exe directly (if it's a 64-bit dump). It will analyze your dump file and write a result json file.

First, yet _NT_SYMBOL_PATH properly in your command shell window. Then invoke it like that SuperDump.exe --dump <path-to-your-dmp-file> --out result.json.

If you get an OOM here, we need to debug this :). If the it runs through and result.json looks valid, then at least the actual dump analysis works.

If that works, try to start SuperDumpService with build/runsuperdump.cmd (it's the actual web-frontend that ultimatively invokes the analysis on uploaded files). But before starting it, edit the cmd file and adapt the line where it says set _NT_SYMBOL_PATH=... to your needs.

For uploading .pdb files within a zip, you additionally need to adapt a setting in appsettings.json. Set SuperDumpSettings::LocalSymbolCache setting to your local symbol cache directory.

[andresod]

@discostu105 thanks for the quick reply.

I'm using the docker-image. Where can I download the release binaries?

This is how I run the container: C:\WINDOWS\system32>docker run -d -p 80:5000 -e "_NT_SYMBOL_PATH=srv*C:\work\sym_cache*\\<our symbol server url>*https://msdl.microsoft.com/download/symbols" -v c:\work\sym_cache:c:\work\sym_cache -v superdump:C:\superdump\data\dumps discostu105/superdump

I use the same setting for _NT_SYMBOL_PATH when using the standalone windbg on my machine locally.

I was wondering if it perhaps has something to do with permissions, i.e. the docker image might not be able to access our symbol server -- it kind of works when I populate the sym_cache folder (same one I bind to in the run cmd) by running the standalone windbg first. However, I have no issues accessing the symbol server using a nanoserver container.

I will try building superdump.exe myself (in case I can't obtain the release binaries) and try your suggestions for 2 and 3.

Regarding the setting for .pdb files within a zip, I assume this requires building the binaries or is there a way to supply this setting via the run cmd?

[discostu105]

You can find Releases here: https://github.com/Dynatrace/superdump/releases

Just published a new version (3.0) with all the latest features included.

[discostu105]

I've also pushed the latest version to docker hub.

Then I tried this command (which IMHO should be correct): docker run -d -p 80:5000 -e "_NT_SYMBOL_PATH=srv*c:\symbols*\\mysymbolserver1\symstore;srv*c:\symbols*\\mysymbolserver2\symstore;srv*c:\symbols*https://msdl.microsoft.com/download/symbols" -v superdump:C:\superdump\data\dumps -v C:\symbols:C:\symbols superdump:dev

However, I'm also having trouble retrieving my private symbols. I think it is an issue that only occurs from within docker images, so a workaround for you might be to go without docker for now.

If you want to dig further into the docker/symbolserver issue, I recommend uploading a dump, then enter the "Interactive Mode". You can go with all sorts of WinDbg commands in there, but it actually runs inside the docker container.

[andresod]

thanks @discostu105 the release binary works much better than the docker image. However, there are still some differences in how windbg reports the results, even though symbol path is identical. In the standalone windbg, I can see details about the source location and line number (FAULTING_SOURCE_LINE/FILE/LINE_NUMBER) when viewing the stack trace (I replaced the actual names with fictional): 00 (Inline Function) -------- -------- mymodule!myfun+0x3f [<path to c source file> @ 355]

in superdump windbg.log it only shows mymodule!myfun+0x96 Unfortunately, I can't give you more details as the interactive mode seems to not be working properly in the release binary, it only shows:

Web Terminal Type 'cls' to clear the console

That is, no windbg interactive session is started. I'm probably missing one of the dependencies.

btw, there was no need for me to upload the pdbs alongside the crashdump file -- superdump was able to find the pdbs via the symbolpath.

[discostu105]

Concerning WinDbg.log, you can look up the actual commands that are issued here https://github.com/Dynatrace/superdump/blob/master/src/SuperDump/Analyzers/WinDbgAnalyzer.cs#L21. Maybe it just needs the proper commands to match what you were doing in WinDbg?

Concerning "inlined" functions: these are not shown in SuperDump reports for some reason, which is a kown issue: https://github.com/Dynatrace/superdump/issues/54

Interactive Mode: Open browser dev-tools (F12) and look for the websocket connection under /cmd. You might be able to see an error message in one of the websocket frames. Most likely, you don't have cdb.exe installed. SuperDump expects it in a certain directory, which needs to be configured here: https://github.com/Dynatrace/superdump/blob/master/conf/appsettings.json.template#L59

[andresod]

thanks again for your prompt response!

I have cdb installed at the standard location, which matches the path in appsettings.json. The error i'm seeing in devtools is: WebSocketManager.js:94 WebSocket connection to 'ws://localhost:5001/cmd' failed: Connection closed before receiving a handshake response Connection.start @ WebSocketManager.js:94 webterm.js:92 Disconnected! I can run cdb just fine from the cmd line.

[solved] after changing both the ASPNETCORE_URLS to http, the interactive sessions are now working.

you are right, cdb seems to be generating different info when using the k command to show stack trace compared to standalone windbg. Either the option -lines needs to be supplied via the cdb command line option, or the cdb command .lines needs to be specified at the beginning of the analysis. The -lines options seems easier since that can be customized by the user. Is there a way to change this setting globally so that not just the interactive session will use this option specified in the appsettings file?

[discostu105]

The websocket/https problem: It's a bug here: https://github.com/Dynatrace/superdump/blob/master/src/SuperDumpService/wwwroot/js/webterm.js#L83

For https, the connection needs to be made to wss://, not ws://. We could change this hard-coded (since https is default anyway), or make it detect if the current connection is https or not and decide the protocol accordingly.

Currently the list of WinDbg commands is hard-coded. Feel free to add that argument to the code, it sounds useful anyway. Another option would be to make the list of commands configurable in a separate file.

[andresod]

by now, this discussion has deviated from the initial report about symbols/pdb files - is it OK to continue here or should I create a new issue? are you proposing I should make a pull request for those two items?

[discostu105]

Sure, feel free to do a PR!

[andresod]

Other than that: a secret feature of SuperDump is: if you upload a zip, which contains .pdb files, they are automatically added to the local symbol cache (inside the container!). Dumps are analyzed after that. So, you could upload zip files, containing dumps and matching pdbs and it should work fine.

It seems that the interactive session is not able to find those pdbs that are packaged, unless I add the following to the sympath: .sympath+ c:\superdump\data\symbols

Obviously, path above is installation dependent. Would it be possible to somehow add this in code using a relative path? Superdump is able to 'find' the pdbs during initial analysis, just not in the interactive session.