is the test suite something that could be factored out/generalized?

[Gankra]

You've done some incredible stuff in there making it possible to generate dumps, use dump_syms, and then load dumps again. I've been wanting a tool that is two binaries, minidump-harness and futility.

futility

(it seems like src/bin/test.rs is basically this)

The futile utility application, providing a "crash suite"

• takes a simple cli arg choosing which func to call that will die in an interesting way
• we can build for whatever platform
• run dump_syms on

minidump-harness

(it seems like tests/ is kinda this)

Orchestrates building/analyzing/torturing futility and storing/diffing the results. it would serve three roles:

• regression tests (running in ci)
• getting a particular intermediate result of the pipeline for inspection
• enabling us to quickly test/diff a change across our entire pipeline

The first is straight-forward and you mostly have that setup (although #30 still needs to be done). The second is more subtle. You can see my initial attempts at this problem with my socc-pair utility which largely automates diffing/benching a local version of minidump-stackwalk against what mozilla has in production. This is great for testing if I fixed a bug in production, but it's still more manual than I'd like.

The Dream

$ minidump-harness run base-trial
<spews a bunch of shit into a dir called base-trial>

<now i make some changes like [patch]ing local changes to rust-minidump/symbolic/dump_syms...>

$ minidump-harness run patched-trial
<spews a bunch of shit into a dir called patched-trial>

$ minidump-harness diff base-trial patched-trial
<spits out a report of what changed>

The Design

minidump-harness run $DIR would:

• build futility, putting the target in $DIR/target
• run dump-syms on it, putting the symbols in $DIR/symbols
• run every crash suite it has, putting the dumps in $DIR/dumps/$suite.dmp
• run minidump-stackwalk --cyborg --verbose=trace --symbols-path=$DIR/symbols on each minidump, putting:
  • json in $DIR/results/$suite.json
  • human in $DIR/results/$suite.txt
  • logs in $DIR/results/$suite.log.txt

minidump-harness diff $DIR1 $DIR2 would:

• spit out a diff of particular details (default would probably be what socc-pair does?)
• but ideally would streamline other diff/filter queries like "did the human output change" or "just tell me about this suite"
• probably also want to be able to cargo-insta (snapshot) some of these results?

---

Right now you have some of these things, but a lot of the test implementations are in the harness and kind of manually adhoced. I don't fully understand your design, and all the constraints there are. I'm happy to do a lot of legwork here, but you've clearly done a lot stuff that's worth using/extending instead of me kicking off a green-field project.

[Gankra]

cc @mstange and @Jake-Shadle

[Jake-Shadle]

I think something like this would be great. The MacOS dump_syms stuff I added was loosely based on the end-to-end testing I did in https://github.com/EmbarkStudios/crash-handling/tree/main/minidumper-test though there I at the moment just doing the basic "does this minidump contain the expected crash reason" without checking the symbolicated stack traces for the thread(s) also match, but that was something I wanted to add eventually.

I guess the one big difference to me is that the dream scenario you specified above feels a bit too manual for me, but maybe this is just a different workflow for doing changes before making automated tests?

[Gankra]

yeah i made the dream slightly manual just so we could do absolutely whatever we want and extract only the parts we care about for individual runs.

i.e. "this old dump_syms version had this problem, I want a sym file showing that for my tests, so I will [patch] symbolic back to that version and do minidump-harness run and grab the sym file it spits out for my test-suite"

making automated tests I think would look closer to cargo insta snapshots, where we have a copy of the outputs we care about saved in one dir, and the tests just minidump-harness run and diff against the snapshot.

[Gankra]

basically it would be useful as a

• test-harness
• artifacts-for-other-tests/projects generator
• debugger

[Gankra]

Ok here's a rough diagram of what a full deployment architecture looks like.

I've broken the architecture into 3 major sections:

• your build
• your client (user)
• your server (you)

The client and server are further broken up into separate processes.

Rough atlas:

• solid arrows: code invoking
• dashed arrows: data flow
• diamond boxes: data (files/structs)
• square boxes: specific rust-minidump/crash-handling crates
• round boxes: abstract services that we don't provide but you "need" to glue things together. I have labeled some of them with mozilla-specific names because those are what I know, but a lot of these could be labeled "Sentry". Many of these are the things a test harness would need to fill in.

Note that Mozilla (and Sentry?) don't use any of the "client" stuff right now except for minidump-writer in some specific situations. Migrating to minidump-writer completely is desirable for Mozilla (kill breakpad), but the other stuff is less clearly important to us.

graph TD;
subgraph build
  cargo
  dump_syms
  http-symbol-sender(http-symbol-sender)
  binary{.exe/.pdb}
  sym{.sym}
end

subgraph client
  subgraph app
    YOUR_ACTUAL_APP(YOUR ACTUAL APP)
    minidumper::Client
    crash-handler
  end

  subgraph crash-monitor
    minidumper::Service
    minidump-writer
    http-dump-sender(http-dump-sender)
    context{CrashContext}
    dump{.dump}
  end

  
end

subgraph server
  subgraph crash-server
    http-dump-receiver(http-dump-receiver)
    minidump-processor
    minidump
    structMinidump{Minidump}
    report{report.json}
  end
  subgraph symbol-server
    tecken(tecken)
    http-symbol-receiver(http-symbol-receiver)
  end
  subgraph crash-stats
    socorro(socorro)
  end
end

cargo -.-> binary
binary -.-> dump_syms
binary -.-> deploy(deploy)
dump_syms -.-> sym
sym -.-> http-symbol-sender

crash-handler ---> minidumper::Client
minidumper::Client ---> minidumper::Service
minidumper::Service ---> minidump-writer
minidump-writer -.-> dump
dump -.-> http-dump-sender
minidumper::Service ---> http-dump-sender 
minidumper::Service -.-> context
context -.-> minidump-writer

http-dump-receiver ---> minidump-processor
http-dump-receiver ---> minidump
minidump -.-> structMinidump
structMinidump -.-> minidump-processor
tecken -. .sym .-> minidump-processor
minidump-processor -.-> report
report -.-> http-dump-receiver
http-dump-receiver -. report.json .-> socorro

http-symbol-sender -. .sym .-> http-symbol-receiver
http-dump-sender -. .dump .-> http-dump-receiver

graph source

graph TD;
subgraph build
  cargo
  dump_syms
  http-symbol-sender(http-symbol-sender)
  binary{.exe/.pdb}
  sym{.sym}
end

subgraph client
  subgraph app
    YOUR_ACTUAL_APP(YOUR ACTUAL APP)
    minidumper::Client
    crash-handler
  end

  subgraph crash-monitor
    minidumper::Service
    minidump-writer
    http-dump-sender(http-dump-sender)
    context{CrashContext}
    dump{.dump}
  end

  
end

subgraph server
  subgraph crash-server
    http-dump-receiver(http-dump-receiver)
    minidump-processor
    minidump
    structMinidump{Minidump}
    report{report.json}
  end
  subgraph symbol-server
    tecken(tecken)
    http-symbol-receiver(http-symbol-receiver)
  end
  subgraph crash-stats
    socorro(socorro)
  end
end

cargo -.-> binary
binary -.-> dump_syms
binary -.-> deploy(deploy)
dump_syms -.-> sym
sym -.-> http-symbol-sender

crash-handler ---> minidumper::Client
minidumper::Client ---> minidumper::Service
minidumper::Service ---> minidump-writer
minidump-writer -.-> dump
dump -.-> http-dump-sender
minidumper::Service ---> http-dump-sender 
minidumper::Service -.-> context
context -.-> minidump-writer

http-dump-receiver ---> minidump-processor
http-dump-receiver ---> minidump
minidump -.-> structMinidump
structMinidump -.-> minidump-processor
tecken -. .sym .-> minidump-processor
minidump-processor -.-> report
report -.-> http-dump-receiver
http-dump-receiver -. report.json .-> socorro

http-symbol-sender -. .sym .-> http-symbol-receiver
http-dump-sender -. .dump .-> http-dump-receiver