spoa mirror crashes after multiple segfaults

[gdriban]

Over time, spoa-mirror crashes and also causes crash of forwarding haproxy.

Using dmesg you can see repeated segfaults from spoa mirror. Example: [10196349.094441] sm/wrk: 6[23039]: segfault at 0 ip 000000000040350f sp 00007fcab955eba0 error 4 in spoa-mirror[400000+10000]

Running version 1.2.12 of spoa-mirror. ./spoa-mirror -V spoa-mirror v1.2.12 [build 2415] by Miroslav Zagorac [email protected], Jul 19 2020 Had been experiencing crashes of haproxy and stopping spoa-mirror seems to have solved the issue.

OS is Oracle Linux 7 stable v7.8.0

This issue was original reported as a piggyback on https://github.com/haproxytech/spoa-mirror/issues/16. Opening new issue. Was requested by @zaga00 to run debug version and get a core file. May not be able to do that because spoa-mirror is only running on production servers. If we can get that done will post core file when we have it.

Thanks.

[zaga00]

Hello @gdriban,

thank you for reporting the bug. I couldn't reproduce the bug so in the next few days I'll take a closer look at what it's all about, now unfortunately I don't have time for that.

[gdriban]

It is an issue that occurs over time and probably only with actual traffic going through the mirror. You'd have to leave it run for several days at minimum. Will let you know of any additional info we can provide when we have it.

[gdriban]

@zaga00 followed instructions for building spoa-mirror in the README. Before I run the binary on one of my production servers, I just want to make certain that performing the build this way will add debug symbols to the binary so that we get a core file on segfault: ./configure --enable-debug make all Thank you.

[zaga00]

Hello @gdriban,

for the debugging process it is most important not to delete symbols (debug information) from the executable (with the 'strip' command). If you also use the '--enable-debug' option when configuring the source before compiling, then the '-O0' optimization level will be used when compiling the source (instead of '-O2' which is used if '--enable-debug' option is not used). Of course, the 'xx' option also allows the use of the '--debug' option when starting the program, as well as a detailed printout of the functions used with parameters within the program itself which can be written to a log file (with the '--logfile' option). In short, however you configure the source, it is only important that you do not strip symbols from the program.