using --symbols-url, stackwalker marks module as missing symbols instead of corrupt on ParseError

[willkg]

The stackwalker can emit corrupt_symbols: true and the code suggests it'll do that if there's a parse error. It'd be handy to have Crash Stats surface corrupt symbols files.

I read through the code and tried some of the things in the comments around where it emits a ParseError and I can't figure out how to build a sym file that triggers a ParseError that results in a corrupt_symbols: true. Do I lack imagination? Is it possible that ParseError gets treated as a missing file?

[mstange]

I was able to get a "corrupt_symbols": true in the "modules" section of the JSON. But in the "threads" / "frames" part of the JSON, the relevant frames had "missing_symbols": true.

  "modules": [
    {
      "base_addr": "0x00000001040e4000",
      "cert_subject": null,
      "code_id": "e2fcb1a819d7349daaaf83c97f3793bc",
      "corrupt_symbols": true,                                         // <-------------- !!!
      "debug_file": "crash-client",
      "debug_id": "E2FCB1A819D7349DAAAF83C97F3793BC0",
      "end_addr": "0x0000000104190000",
      "filename": "crash-client",
      "loaded_symbols": true,
      "missing_symbols": false,
      "symbol_url": null,
      "version": null
    },
  // ...
  "thread_count": 11,
  "threads": [
    {
      "frame_count": 11,
      "frames": [
        {
          "file": null,
          "frame": 0,
          "function": null,
          "function_offset": null,
          "inlines": null,
          "line": null,
          "missing_symbols": true,
          "module": "crash-client",
          "module_offset": "0x0000000000013f9c",
          "offset": "0x00000001040f7f9c",
          "trust": "context",
          "unloaded_modules": null
        },

To get this output, I did the following:

1. I used minidump-pipeline to generate a bunch of dmp and sym files (in the minidump-pipeline/runs/run/ directory).
2. I edited crash-client.sym by copying the MODULE line further down into the file, in order to hit this ParseError.
3. I ran minidump-stackwalk by executing the following in this repo:

cargo run --release -p minidump-stackwalk -- --json --pretty '/Users/mstange/code/minidump-pipeline/runs/run/dumps/segv.dmp' --symbols-path '/Users/mstange/code/minidump-pipeline/runs/run/syms'

[willkg]

Interesting! I had seen that ParseError code and had tried copying the MODULE line a few lines down into the FILE, but didn't get a ParseError. However, I was serving the sym file locally with a symbol server.

If I switch to running the stackwalker and passing in a path to the sym file instead of using a symbol server, the stackwalker gets the ParseError and I see the coveted corrupt_symbols: true in the output.

Maybe when the symbol supplier is a web server and the stackwalker gets the ParseError, it bails and treats it as a missing symbol? If so, then we'll never get a corrupt_symbols: true in crash ingestion.

[willkg]

I think this is a bug. Steps to reproduce:

1. take an existing symbol file that's needed for a dump you have and copy the MODULE line to later in the file
2. run the stackwalker using --symbol-path /path/to/symbols and observe "corrupt_symbols": true in json output for the module
3. set up a symbol server to serve the directory and using --symbol-url http://serverurl/ and observe "corrupt_symbols": false in json output for the module

For the symbol server, I used the Python built-in HTTP server, but I had to gunzip the sym file so it was served as text/plain first.

I'm not sure it's a big deal, but it definitely makes it harder for us to know when we have corrupted symbols files on the symbols server.

[mstange]

Ah, so this explains why the try crash report from this bugzilla comment displayed XUL as missing symbols: The symbols were there, but they were using the inline-ful sym format before rust-minidump was able to parse it, so parsing failed, and due to this bug we marked symbols as missing instead of corrupt.