getsentry
symbolicli quietly fails to symbolize if cabextract is not found in PATH
I'm trying to use symbolicli to analyze minidumps in offline mode and it only symbolizes very few symbols in the stacktraces, the rest come back unresolved.
Environment
Commit 271386c45887 built with cargo build --release on Fedora 37 x86_64.
Steps to Reproduce
My ~/.symboliclirc:
cache_dir = "/home/ts/.symbolicli/cache"
[[sources]]
id = "company"
type = "http"
url = "https://dev.my-company.com/symbols"
layout = { type = "symstore_index2", casing = "default" }
[[sources]]
id = "microsoft"
type = "http"
url = "https://msdl.microsoft.com/download/symbols"
Command executed:
target/release/symbolicli --log-level=info --offline --format=pretty ~/Downloads/general2d-test2-session-ezhiZjhl-pid-820-user-jenkins-1.dmp
Expected Result
I hoped to get a stacktrace back with function names resolved
Actual Result
Functions are mostly not resolved, with a few exceptions:
[ts@spark symbolicator]$ target/release/symbolicli --log-level=info --offline --format=pretty ~/Downloads/general2d-test2-session-ezhiZjhl-pid-820-user-jenkins-1.dmp
2023-05-01T10:38:12.889493Z INFO symbolicli: successfully parsed local event
2023-05-01T10:38:12.889527Z INFO symbolicli: symbolicating minidump
Frame #0
Trust: context
Instruction: 0x7ffdbadd0cf1
Module: Qt5Widgets.dll +0x40cf1
Frame #1
Trust: scan
Instruction: 0x7ffdbadd67b4
Module: Qt5Widgets.dll +0x467b4
Frame #2
Trust: scan
Instruction: 0x7ffe180efde5
Module: ucrtbase.dll +0xfde5
Function: malloc_base + 0x35
Frame #3
Trust: cfi
Instruction: 0xffffffff
Frame #4
Trust: scan
Instruction: 0x7ffdbadca5ec
Module: Qt5Widgets.dll +0x3a5ec
Frame #5
Trust: scan
Instruction: 0x7ffdbadc16e6
Module: Qt5Widgets.dll +0x316e6
Frame #6
Trust: scan
Instruction: 0x7ffdbadc1465
Module: Qt5Widgets.dll +0x31465
Frame #7
Trust: scan
Instruction: 0x7ffdda641244
Module: Qt5QuickWidgets.dll +0x1244
Frame #8
Trust: scan
Instruction: 0x7ffdba9e4cf7
Module: Qt5WebEngineWidgets.dll +0x14cf7
Frame #9
Trust: scan
Instruction: 0x7ffdba9e747f
Module: Qt5WebEngineWidgets.dll +0x1747f
Frame #10
Trust: scan
Instruction: 0x7ffda613c560
Module: Qt5WebEngineCore.dll +0x556c560
Frame #11
Trust: scan
Instruction: 0x7ffdba9d59b9
Module: Qt5WebEngineWidgets.dll +0x59b9
...
...
Frame #38
Trust: cfi
Instruction: 0x1dcf2390000
Frame #39
Trust: scan
Instruction: 0x7ffe1a95b7ff
Module: ntdll.dll +0x2b7ff
Function: RtlpLowFragHeapAllocFromContext + 0x1af
Frame #40
Trust: cfi
Instruction: 0x1dcd7a2d190
...
...
Frame #53
Trust: scan
Instruction: 0x7ffda64ca740
Module: Qt5WebEngineCore.dll +0x58fa740
Frame #54
Trust: scan
Instruction: 0x7ffe180efde5
Module: ucrtbase.dll +0xfde5
Function: malloc_base + 0x35
Frame #55
Trust: cfi
Instruction: 0x7ffda4b7e7af
Module: Qt5WebEngineCore.dll +0x3fae7af
Frame #56
Trust: scan
Instruction: 0x7ffe1a955ba0
Module: ntdll.dll +0x25ba0
Function: RtlpFreeHeapInternal + 0x490
Using the same symbol servers Visual Studio 2022 symbolizes the same minidump correctly.
I had a look in the files in the cache directory and a bunch of them report a weird looking error. I think this may be the cause of the problem?
[ts@spark cache]$ grep -r malformed
objects/v1/b0/6748cb/e13484f30cb8f48e61061899b0d8cf1e1598f803168a48fa845d18c4:malformedNo such file or directory (os error 2)
objects/v1/fe/786073/a671e3740cb068b47c7331feb57b10ba55c4a495e7cc1338828edf8f:malformedNo such file or directory (os error 2)
objects/v1/bb/f7115d/903c7c55ca0e56f9cfbc827ef0f25cb7fcedfeb42ee070d570cf549f:malformedNo such file or directory (os error 2)
objects/v1/61/41b795/384cef38654142b39a4b604beea40e2db0783265f523d2ddc380e90d:malformedNo such file or directory (os error 2)
objects/v1/7b/e672a2/af096997ee5cdba7d620e21052734ca856b45a74a915497c0c88ebbb:malformedNo such file or directory (os error 2)
objects/v1/57/dfb68e/6fbd0d6a466ca90e26be17c5c626204be672a64ddd05f9afaec2dada:malformedNo such file or directory (os error 2)
objects/v1/57/a1196c/ac3c25e3185a4e0637f71d0c36e078dff629c82ec835317f35a8a3d5:malformedNo such file or directory (os error 2)
objects/v1/65/ff4a37/08fa05c12f50bffa8353d22871ce218fbc70f07376614c96431b46cc:malformedNo such file or directory (os error 2)
objects/v1/c5/2642b9/e6e6297508d52025a4e6ffc0d08c4cc4e2a15f781d5806ccf74d7db9:malformedNo such file or directory (os error 2)
object_meta/v1/b0/6748cb/e13484f30cb8f48e61061899b0d8cf1e1598f803168a48fa845d18c4:malformedNo such file or directory (os error 2)
object_meta/v1/fe/786073/a671e3740cb068b47c7331feb57b10ba55c4a495e7cc1338828edf8f:malformedNo such file or directory (os error 2)
object_meta/v1/bb/f7115d/903c7c55ca0e56f9cfbc827ef0f25cb7fcedfeb42ee070d570cf549f:malformedNo such file or directory (os error 2)
object_meta/v1/61/41b795/384cef38654142b39a4b604beea40e2db0783265f523d2ddc380e90d:malformedNo such file or directory (os error 2)
object_meta/v1/7b/e672a2/af096997ee5cdba7d620e21052734ca856b45a74a915497c0c88ebbb:malformedNo such file or directory (os error 2)
object_meta/v1/57/dfb68e/6fbd0d6a466ca90e26be17c5c626204be672a64ddd05f9afaec2dada:malformedNo such file or directory (os error 2)
object_meta/v1/57/a1196c/ac3c25e3185a4e0637f71d0c36e078dff629c82ec835317f35a8a3d5:malformedNo such file or directory (os error 2)
object_meta/v1/65/ff4a37/08fa05c12f50bffa8353d22871ce218fbc70f07376614c96431b46cc:malformedNo such file or directory (os error 2)
object_meta/v1/c5/2642b9/e6e6297508d52025a4e6ffc0d08c4cc4e2a15f781d5806ccf74d7db9:malformedNo such file or directory (os error 2)
The accompanying .txt file in the cache to one of those reporting malformed above points to a debug source file that exists:
[ts@spark cache]$ cat object_meta/v1/c5/2642b9/e6e6297508d52025a4e6ffc0d08c4cc4e2a15f781d5806ccf74d7db9
malformedNo such file or directory (os error 2)[ts@spark cache]$ cat object_meta/v1/c5/2642b9/e6e6297508d52025a4e6ffc0d08c4cc4e2a15f781d5806ccf74d7db9.txt
scope: global
source: company
location: https://dev.my-company.com/symbols/Qt/Qt5WebEngineCore.dll/63EA5B206c6a000/Qt5WebEngineCore.dl_
Could this possibly be related to the use of layout type "symstore_index2"?
I've found in crates/symbolicator-service/src/caching/fs.rs:
// States a cache item can be in:
// * negative/empty: An empty file. Represents a failed download. mtime is used to indicate
// when the failed download happened (when the file was created)
// * malformed: A file with the content `b"malformed"`. Represents a failed symcache
// conversion. mtime indicates when we attempted to convert.
// * ok (don't really have a name): File has any other content, mtime is used to keep track
// of last use.
So I guess I have a failed symcache conversion. I have run symbolicli with --log-level=trace without seeing any log messages about failed symcache conversion, so I am still not sure how to troubleshoot this issue.
I tried rebuilding symbolicator today on a different machine and all of a sudden the problem is gone. It now is able to symbolize the minidump correctly. I'll close the issue.
Back on the machine where I first had this problem it still occurs with the same build and I did some troubleshooting and managed to identify the problem. symbolicli tries to run cabextract and if that program is not found in the PATH symbolization quietly fails and the only evidence is "malformedNo such file or directory (os error 2)" in the cache files. I'll reopen the issue - symbolicli should emit a message at log-level warning or error that cabextract was not found.
The fix would be to improve maybe_decompress_file to
- first check if
cabextractis available - return a better error on failure.
