dragonflydb
investigate minicore dumper integration with dragonfly
Corefiles are important so we could iterate and fix datarace/segfault bugs
Unfortunately with in-memory stores, linux-created corefiles can reach dozens, hundreds of gigabytes.
I would like to explore two alternatives:
- https://github.com/Percona-Lab/coredumper/
- https://github.com/google/breakpad/
These are the challenges I see with coredumpers:
- ARM64 support is a must
- Fibers support is a must. Our fibers allocate their stack on heap and it's not immediately clear if any of the coredumpers will work out of the box.
The goal is to investigate what can be the simplest and the least intrusive way to integrate mini-coredumper into Dragonfly.
Also, there is this one: https://github.com/diamon/minicoredumper
Its library is licensed under https://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html but it seems that LGPL allows us linking the library and keep the BUSD license we have in place.
Another caveat: we saw problems with using a piped /proc/sys/kernel/core_pattern when a proactor thread crashes. In other words when a kill signal is sent to a ProactorXX thread and not a "main" thread. We have not investigated it further.
The last one looks promising: https://tracingsummit.org/ts/2016/files/Ogness_minicoredumper.pdf
minicoredumper produces small files but it only records stack variables, and if they point to heap regions - those will be zeroed out. so any data member besides stack will have 0. It is possible to whitelist specific regions/variables by providing lengths of how much data to store from there. see https://www.linutronix.de/minicoredumper/man/man5/minicoredumper.recept.json.5.html
it is better than nothing for sure, the core files are very small but maybe we can get something better.