Crash when talking

[Krzmbrzl]

Description

When talking, Mumble crashes every now and then

Steps to reproduce

1. Make conversation in Mumble

Mumble version

0cef808148669a20d3aa70145ab45394a8fcdc63

Mumble component

Client

OS

Linux

Reproducible?

Yes

Additional information

Probably related to https://github.com/mumble-voip/mumble/pull/5352

Relevant log output

Backtrace:

[Switching to Thread 0x7fffa67fc700 (LWP 10250)]
0x00007ffff6d78605 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
(gdb) bt
#0  0x00007ffff6d78605 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#1  0x00007ffff6e5551a in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#2  0x00007ffff6e615a9 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#3  0x000055555594cc8b in CryptStateOCB2::ocb_decrypt (this=0x7fff90007240, 
    encrypted=0x7fffa67fa5d4 "\252\006\070W\366cO\344\206\330[\300\335\334<B\312\361 ͆\267w\213y6\211\310\307\366#I꧀u#X\345\027\346\350\262@\027\233v8\311\357\367!\201\220~\324\326b\334r{)\361W\374WoMt\325\005|\020Ƣ\032\026\374\027\037\233\362e\331ߌ?\351\364\324\037\370\t\177\320}G\357k\232\264\213\330\370vl\237So\374:\034\a\343g\017t\252{\"\006X%?ǜ\337H\334j|\374\205\025\264\356\341\214\313\023P\250W\035\355\026\205\210)\237/\340\263<\326\025`\233\"`f\005.\300\322R\201\372\225\265\261\362\372\227S\374\306\065\336}[f\317J\005\031(y>G\273i\004?\022\336@"..., plain=0x7fffa67fadd0 "", len=245, 
    nonce=0x7fff90007299 "\223\217\221,\001\322SX\026)㫗<\375#", '\217' <repeats 147 times>, '\216' <repeats 37 times>..., tag=0x7fffa67fa510 "0\245\177\246\377\177")
    at /home/Krzmbrzl/Documents/Git/mumble/src/crypto/CryptStateOCB2.cpp:354
#4  0x000055555594c3d8 in CryptStateOCB2::decrypt (this=0x7fff90007240, 
    source=0x7fffa67fa5d0 "\223\236\025N\252\006\070W\366cO\344\206\330[\300\335\334<B\312\361 ͆\267w\213y6\211\310\307\366#I꧀u#X\345\027\346\350\262@\027\233v8\311\357\367!\201\220~\324\326b\334r{)\361W\374WoMt\325\005|\020Ƣ\032\026\374\027\037\233\362e\331ߌ?\351\364\324\037\370\t\177\320}G\357k\232\264\213\330\370vl\237So\374:\034\a\343g\017t\252{\"\006X%?ǜ\337H\334j|\374\205\025\264\356\341\214\313\023P\250W\035\355\026\205\210)\237/\340\263<\326\025`\233\"`f\005.\300\322R\201\372\225\265\261\362\372\227S\374\306\065\336}[f\317J\005\031(y>G\273i\004"..., dst=0x7fffa67fadd0 "", crypted_length=249) at /home/Krzmbrzl/Documents/Git/mumble/src/crypto/CryptStateOCB2.cpp:196
#5  0x00005555557f0d2a in ServerHandler::udpReady (this=0x555556c08340) at /home/Krzmbrzl/Documents/Git/mumble/src/mumble/ServerHandler.cpp:229
#6  0x00005555555fb110 in ServerHandler::qt_static_metacall (_o=0x555556c08340, _c=QMetaObject::InvokeMetaMethod, _id=11, _a=0x7fffa67fb720)
    at /home/Krzmbrzl/Documents/Git/mumble/build/src/mumble/mumble_autogen/EWIEGA46WW/moc_ServerHandler.cpp:143
#7  0x00007ffff606de00 in ?? () from /lib/x86_64-linux-gnu/libQt5Core.so.5
#8  0x00007ffff6c17ba4 in ?? () from /lib/x86_64-linux-gnu/libQt5Network.so.5
#9  0x00007ffff6c17c39 in ?? () from /lib/x86_64-linux-gnu/libQt5Network.so.5
#10 0x00007ffff6c29f49 in ?? () from /lib/x86_64-linux-gnu/libQt5Network.so.5
#11 0x00007ffff78ebdc3 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /lib/x86_64-linux-gnu/libQt5Widgets.so.5
#12 0x00007ffff78f4bb8 in QApplication::notify(QObject*, QEvent*) () from /lib/x86_64-linux-gnu/libQt5Widgets.so.5
#13 0x00007ffff6036daa in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /lib/x86_64-linux-gnu/libQt5Core.so.5
#14 0x00007ffff6092205 in ?? () from /lib/x86_64-linux-gnu/libQt5Core.so.5
#15 0x00007ffff561417d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#16 0x00007ffff5614400 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#17 0x00007ffff56144a3 in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#18 0x00007ffff6091602 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007ffff60358ab in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /lib/x86_64-linux-gnu/libQt5Core.so.5
#20 0x00007ffff5e4f2c2 in QThread::exec() () from /lib/x86_64-linux-gnu/libQt5Core.so.5
#21 0x00005555557f2364 in ServerHandler::run (this=0x555556c08340) at /home/Krzmbrzl/Documents/Git/mumble/src/mumble/ServerHandler.cpp:453
#22 0x00007ffff5e5045c in ?? () from /lib/x86_64-linux-gnu/libQt5Core.so.5
#23 0x00007ffff735f609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#24 0x00007ffff5964293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Screenshots

No response

[Krzmbrzl]

@TerryGeng do you happen to have an idea on this one?

[TerryGeng]

😅 Wasn't expecting a new version of OpenSSL would cause such a mess.

[Green-Sky]

got the same crash

#0  0x00007ffff6e6d5a6 in  () at /lib/x86_64-linux-gnu/libcrypto.so.1.1
#1  0x0000555555895c75 in CryptStateOCB2::ocb_decrypt(unsigned char const*, unsigned char*, unsigned int, unsigned char const*, unsigned char*)
    (this=this@entry=0x7fffa002a8e0, encrypted=encrypted@entry=0x7fffa7ffd634 "...", len=606, nonce=nonce@entry=0x7fffa002a939 "...", 'H' <repeats 139 times>, 'G' <repeats 45 times>..., tag=tag@entry=0x7fffa7ffd530 "") at /home/green/workspace/mumble/src/crypto/CryptStateOCB2.cpp:354

will try the rp now. also, i get the crash "only" every couple of days, so feedback might come late... edit: commit was f16b47c81

[TerryGeng]

got the same crash

#0  0x00007ffff6e6d5a6 in  () at /lib/x86_64-linux-gnu/libcrypto.so.1.1
#1  0x0000555555895c75 in CryptStateOCB2::ocb_decrypt(unsigned char const*, unsigned char*, unsigned int, unsigned char const*, unsigned char*)
    (this=this@entry=0x7fffa002a8e0, encrypted=encrypted@entry=0x7fffa7ffd634 "...", len=606, nonce=nonce@entry=0x7fffa002a939 "...", 'H' <repeats 139 times>, 'G' <repeats 45 times>..., tag=tag@entry=0x7fffa7ffd530 "") at /home/green/workspace/mumble/src/crypto/CryptStateOCB2.cpp:354

will try the rp now. also, i get the crash "only" every couple of days, so feedback might come late... edit: commit was f16b47c

Maybe your PC is great and two threads almost never run on top of each other. :)

Thanks for help us testing!

[Toasterbirb]

Is this problem fixed in the 1.4.230 release? I recently switched to -march=native on my gentoo system, did some updates and recompiled the entire system and now mumble keeps crashing. The crashes are really random and looked similar to this issue

I haven't managed to really get that much information about the problem, mostly just segfaults with no additional information. GDB mentioned libcrypto.so, so I thought it might have something to do with OpenSSL.

[Krzmbrzl]

This issue should not even have existed in 1.4.230 as the changes causing the issue were (afaik) never part of 1.4.230. They are only part of the 1.5 branch (current master) :thinking:

[Toasterbirb]

I'll try to get a bit more information about the segfaults then. Maybe this is a new issue

[Toasterbirb]

Waited in a call alone and nothing happened. Then someone joined in like about an hour later and it crashed in just a few minutes.

This is the output I got in gdb (I have translated some parts of it to English)

[New Thread 0x7fffabfff640 (LWP 24792)]
[Thread 0x7fffd19fe640 (LWP 24730) exited]
[Thread 0x7fffd21ff640 (LWP 24729) exited]
[Thread 0x7fffd36ef640 (LWP 24728) exited]
[Thread 0x7fffd3ef0640 (LWP 24727) exited]
[Thread 0x7fffc568d640 (LWP 24733) exited]
[Thread 0x7fffc4e8c640 (LWP 24734) exited]
malloc(): unaligned tcache chunk detected

Thread 4 "threaded-ml" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff13c2640 (LWP 24725)]
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44      pthread_kill.c: File or directory doesn't exist.

Edit: Another crash just a minute or two later

Thread 4 "threaded-ml" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff13c2640 (LWP 26341)]
0x00007ffff6e08373 in EVP_EncryptFinal_ex () from /usr/lib64/libcrypto.so.1.1

Edit2: Got a few more crashes and some new output

Thread 14 "ServerHandler" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffabfff640 (LWP 29723)]
0x00007ffff6e09903 in EVP_CIPHER_CTX_block_size () from /usr/lib64/libcrypto.so.1.1

[Krzmbrzl]

Could you build a debug version, attach gdb and then print out a full backtrace after the crash?

[Toasterbirb]

I actually already have it built with debug symbols etc. and run it in gdb. Just forgot to get the backtrace :P

I'll wait for another crash

[Toasterbirb]

Here ya go

(gdb) backtrace
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff5a125df in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007ffff59c6a02 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff59b1469 in __GI_abort () at abort.c:79
#4  0x00007ffff5a06888 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff5b3a623 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#5  0x00007ffff5a1c1da in malloc_printerr (str=str@entry=0x7ffff5b3d6f0 "malloc(): unaligned tcache chunk detected") at malloc.c:5664
#6  0x00007ffff5a205ec in tcache_get (tc_idx=<optimized out>) at malloc.c:3195
#7  __GI___libc_malloc (bytes=264) at malloc.c:3313
#8  0x00007ffff6e1950a in CRYPTO_zalloc () from /usr/lib64/libcrypto.so.1.1
#9  0x00007ffff6e08f7b in EVP_CipherInit_ex () from /usr/lib64/libcrypto.so.1.1
#10 0x000055555584fc2d in CryptStateOCB2::ocb_encrypt(unsigned char const*, unsigned char*, unsigned int, unsigned char const*, unsigned char*, bool) ()
#11 0x000055555584fdf6 in CryptStateOCB2::encrypt(unsigned char const*, unsigned char*, unsigned int) ()
#12 0x0000555555766b3d in ServerHandler::sendMessage(char const*, int, bool) ()
#13 0x00005555556638c3 in AudioInput::flushCheck(QByteArray const&, bool, int) ()
#14 0x0000555555664391 in AudioInput::encodeAudioFrame(AudioChunk) ()
#15 0x000055555566542b in AudioInput::addEcho(void const*, unsigned int) ()
#16 0x000055555581ccad in PulseAudioSystem::read_callback(pa_stream*, unsigned long, void*) ()
#17 0x00007ffff1485ded in ?? () from /usr/lib64/libpulse.so
#18 0x00007ffff141215a in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#19 0x00007ffff1414e00 in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#20 0x00007ffff14151f6 in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#21 0x00007ffff1415ad2 in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#22 0x00007ffff149cc49 in pa_mainloop_dispatch () from /usr/lib64/libpulse.so
#23 0x00007ffff149cf86 in pa_mainloop_iterate () from /usr/lib64/libpulse.so
#24 0x00007ffff149d030 in pa_mainloop_run () from /usr/lib64/libpulse.so
#25 0x00007ffff14aba09 in ?? () from /usr/lib64/libpulse.so
#26 0x00007ffff142839e in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#27 0x00007ffff5a1085a in start_thread (arg=<optimized out>) at pthread_create.c:442
#28 0x00007ffff5a93a5c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)

[Krzmbrzl]

hm.... this is 1.4.230, right? Could you just check to see if the issue persists in current master?

[Toasterbirb]

I tried the current master branch and it crashed almost instantly

Fatal (internal) error in /home/toasterbirb/git/mumble/3rdparty/opus/celt/celt_decoder.c, line 118: assertion failed: st->mode == opus_custom_mode_create(48000, 960, NULL)

Thread 8 "threaded-ml" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff11d1640 (LWP 14651)]
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44      pthread_kill.c: File or directory doesn't exist.
(gdb) backtrace
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff572c5df in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007ffff56e0a02 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff56cb469 in __GI_abort () at abort.c:79
#4  0x00007ffff13aba9f in  () at /home/toasterbirb/git/mumble/build/libopus.so.0
#5  0x00007ffff13b76fd in validate_celt_decoder () at /home/toasterbirb/git/mumble/build/libopus.so.0
#6  0x00007ffff13b7800 in celt_decode_with_ec () at /home/toasterbirb/git/mumble/build/libopus.so.0
#7  0x00007ffff13752bc in opus_decode_frame () at /home/toasterbirb/git/mumble/build/libopus.so.0
#8  0x00007ffff1376a13 in opus_decode_native () at /home/toasterbirb/git/mumble/build/libopus.so.0
#9  0x00007ffff1376c34 in opus_decode_float () at /home/toasterbirb/git/mumble/build/libopus.so.0
#10 0x00005555556bf9ec in AudioOutputSpeech::prepareSampleBuffer(unsigned int) ()
#11 0x00005555556b8501 in AudioOutput::mix(void*, unsigned int) ()
#12 0x00005555558275e6 in PulseAudioSystem::write_callback(pa_stream*, unsigned long, void*) ()
#13 0x00007ffff12ded36 in  () at /usr/lib64/libpulse.so.0
#14 0x00007ffff126927f in pa_pdispatch_run () at /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#15 0x00007ffff12c009f in  () at /usr/lib64/libpulse.so.0
#16 0x00007ffff126bd67 in  () at /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#17 0x00007ffff126ee00 in  () at /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#18 0x00007ffff126f1f6 in  () at /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#19 0x00007ffff126fad2 in  () at /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#20 0x00007ffff12d6c49 in pa_mainloop_dispatch () at /usr/lib64/libpulse.so.0
#21 0x00007ffff12d6f86 in pa_mainloop_iterate () at /usr/lib64/libpulse.so.0
#22 0x00007ffff12d7030 in pa_mainloop_run () at /usr/lib64/libpulse.so.0
#23 0x00007ffff12e5a09 in  () at /usr/lib64/libpulse.so.0
#24 0x00007ffff128239e in  () at /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#25 0x00007ffff572a85a in start_thread (arg=<optimized out>) at pthread_create.c:442
#26 0x00007ffff57ada5c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Edit: It seems to crash immediately after I join a server

[Krzmbrzl]

Ah you seem to suffer from the mysterious Opus crash that is encountered by some. See also https://github.com/mumble-voip/mumble/issues/5302

That's of course unfortunate as it doesn't allow us to investigate the previous issue 👀

[Toasterbirb]

Still quite odd indeed. Sometimes mumble wouldn't crash for hours and some days it would crash 3 times in a row in mere seconds (in the 1.4.230 version). The error seems to be somewhat different since the master branch crashes immediately when I join a call

Also the crashes sometimes mentioned libcrypt.so in 1.4.230

[Krzmbrzl]

@TerryGeng could you have a look at the original backtrace for the 1.4.230 crash? Maybe you have an idea what could be the issue...

[Toasterbirb]

I also should probably note that the crash message isn't always the same. It rotates about 3 different error messages. I'll try to get backtraces for all of them.

[Toasterbirb]

Here's one

malloc(): unaligned tcache chunk detected

Thread 4 "threaded-ml" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff13c2640 (LWP 31517)]
0x00007ffff6e08d47 in EVP_CipherInit_ex () from /usr/lib64/libcrypto.so.1.1
(gdb) backtrace
#0  0x00007ffff6e08d47 in EVP_CipherInit_ex () from /usr/lib64/libcrypto.so.1.1
#1  0x000055555584f8f5 in CryptStateOCB2::ocb_encrypt(unsigned char const*, unsigned char*, unsigned int, unsigned char const*, unsigned char*, bool) ()
#2  0x000055555584fdf6 in CryptStateOCB2::encrypt(unsigned char const*, unsigned char*, unsigned int) ()
#3  0x0000555555766b3d in ServerHandler::sendMessage(char const*, int, bool) ()
#4  0x00005555556638c3 in AudioInput::flushCheck(QByteArray const&, bool, int) ()
#5  0x0000555555664391 in AudioInput::encodeAudioFrame(AudioChunk) ()
#6  0x000055555566542b in AudioInput::addEcho(void const*, unsigned int) ()
#7  0x000055555581ccad in PulseAudioSystem::read_callback(pa_stream*, unsigned long, void*) ()
#8  0x00007ffff1485ded in ?? () from /usr/lib64/libpulse.so
#9  0x00007ffff141215a in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#10 0x00007ffff1414e00 in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#11 0x00007ffff14151f6 in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#12 0x00007ffff1415ad2 in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#13 0x00007ffff149cc49 in pa_mainloop_dispatch () from /usr/lib64/libpulse.so
#14 0x00007ffff149cf86 in pa_mainloop_iterate () from /usr/lib64/libpulse.so
#15 0x00007ffff149d030 in pa_mainloop_run () from /usr/lib64/libpulse.so
#16 0x00007ffff14aba09 in ?? () from /usr/lib64/libpulse.so
#17 0x00007ffff142839e in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#18 0x00007ffff5a1085a in start_thread (arg=<optimized out>) at pthread_create.c:442
#19 0x00007ffff5a93a5c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[Toasterbirb]

Here's another

Thread 4 "threaded-ml" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff13c2640 (LWP 15422)]
0x00007ffff6e09950 in EVP_CIPHER_flags () from /usr/lib64/libcrypto.so.1.1
(gdb) backtrace
#0  0x00007ffff6e09950 in EVP_CIPHER_flags () from /usr/lib64/libcrypto.so.1.1
#1  0x00007ffff6df8309 in ?? () from /usr/lib64/libcrypto.so.1.1
#2  0x00007ffff6e08dbf in EVP_CipherInit_ex () from /usr/lib64/libcrypto.so.1.1
#3  0x000055555584f7de in CryptStateOCB2::ocb_encrypt(unsigned char const*, unsigned char*, unsigned int, unsigned char const*, unsigned char*, bool) ()
#4  0x000055555584fdf6 in CryptStateOCB2::encrypt(unsigned char const*, unsigned char*, unsigned int) ()
#5  0x0000555555766b3d in ServerHandler::sendMessage(char const*, int, bool) ()
#6  0x00005555556638c3 in AudioInput::flushCheck(QByteArray const&, bool, int) ()
#7  0x0000555555664391 in AudioInput::encodeAudioFrame(AudioChunk) ()
#8  0x000055555566542b in AudioInput::addEcho(void const*, unsigned int) ()
#9  0x000055555581ccad in PulseAudioSystem::read_callback(pa_stream*, unsigned long, void*) ()
#10 0x00007ffff1485ded in ?? () from /usr/lib64/libpulse.so
#11 0x00007ffff141215a in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#12 0x00007ffff1414e00 in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#13 0x00007ffff14151f6 in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#14 0x00007ffff1415ad2 in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#15 0x00007ffff149cc49 in pa_mainloop_dispatch () from /usr/lib64/libpulse.so
#16 0x00007ffff149cf86 in pa_mainloop_iterate () from /usr/lib64/libpulse.so
#17 0x00007ffff149d030 in pa_mainloop_run () from /usr/lib64/libpulse.so
#18 0x00007ffff14aba09 in ?? () from /usr/lib64/libpulse.so
#19 0x00007ffff142839e in ?? () from /usr/lib64/pulseaudio/libpulsecommon-15.99.so
#20 0x00007ffff5a1085a in start_thread (arg=<optimized out>) at pthread_create.c:442
#21 0x00007ffff5a93a5c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[Toasterbirb]

I got this kind of notification and was kicked out from the call for a sec. First time seeing this one.

The first three lines say:

Server connection was lost
Server connection failed
Error reading: error:

I have also found some correlation with high CPU usage and the crashes. Mumble seems to crash less when I don't have anything CPU intensive open

[Toasterbirb]

Got a new error

Thread 14 "ServerHandler" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffabfff640 (LWP 25729)]
0x00007ffff6d29605 in ?? () from /usr/lib64/libcrypto.so.1.1
(gdb) backtrace
#0  0x00007ffff6d29605 in ?? () from /usr/lib64/libcrypto.so.1.1
#1  0x00007ffff6dfb626 in ?? () from /usr/lib64/libcrypto.so.1.1
#2  0x00007ffff6e080fb in ?? () from /usr/lib64/libcrypto.so.1.1
#3  0x000055555584fec6 in CryptStateOCB2::ocb_decrypt(unsigned char const*, unsigned char*, unsigned int, unsigned char const*, unsigned char*) ()
#4  0x0000555555850496 in CryptStateOCB2::decrypt(unsigned char const*, unsigned char*, unsigned int) ()
#5  0x000055555576956c in ServerHandler::udpReady() ()
#6  0x00007ffff614472e in ?? () from /usr/lib64/libQt5Core.so.5
#7  0x00007ffff6bf1c6f in ?? () from /usr/lib64/libQt5Network.so.5
#8  0x00007ffff6c04fa9 in ?? () from /usr/lib64/libQt5Network.so.5
#9  0x00007ffff78d296f in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib64/libQt5Widgets.so.5
#10 0x00007ffff6110fc8 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib64/libQt5Core.so.5
#11 0x00007ffff6163bee in ?? () from /usr/lib64/libQt5Core.so.5
#12 0x00007ffff4d6436b in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
#13 0x00007ffff4d64628 in ?? () from /usr/lib64/libglib-2.0.so.0
#14 0x00007ffff4d646df in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#15 0x00007ffff6163184 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/libQt5Core.so.5
#16 0x00007ffff610f9bb in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/libQt5Core.so.5
#17 0x00007ffff5f5a72a in QThread::exec() () from /usr/lib64/libQt5Core.so.5
#18 0x0000555555768f1b in ServerHandler::run() ()
#19 0x00007ffff5f5b8c9 in ?? () from /usr/lib64/libQt5Core.so.5
#20 0x00007ffff5a1085a in start_thread (arg=<optimized out>) at pthread_create.c:442
#21 0x00007ffff5a93a5c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)

Just let me know when to stop listing these, they don't seem to have an end :D

[Krzmbrzl]

I think this might be a general memory corruption that causes crashes in more or less random locations. In this case, the backtraces won't help all that much, I am afraid :thinking:

[Toasterbirb]

OpenSSL just got an update to version 1.1.1o and didn't seem to change anything. Got a crash in a few minutes so OpenSSL is probably ruled out unless the problem wasn't fixed in the update or whatever.

I'm beginning to suspect that my switch to -march=native is the cause and I should go back to setting it to the previous -march=znver3 value that I had in make.conf. AFAIK it has a bit less compiling flags. I might just manually revert the mumble dependencies back to -march=znver3 and recompile all of those hoping for the best

[TerryGeng]

I'm very confused while going through the backtrace, since every time it crashes at a different location. (Perhaps having a greater chance dies inside OpenSSL).

@Toasterbirb By the way, at this moment I think we are compiling against OpenSSL 3.0. Can you try updating your OpenSSL to this version?

[Toasterbirb]

@TerryGeng I might be able to do that but it looks a bit risky

All of the 3.x versions are still in testing and don't seem to be fully compatible yet. I'll give it a try and reverse it if something goes wrong

Edit: I attempted the upgrade with both versions (3.0.2 and 3.0.3) but packages dev-libs/pkcs11-helper-1.27.0-r1 and dev-libs/poco-1.10.1 failed compiling. It looks like I have to stay with the latest stable version of OpenSSL for now. Actually even 1.1.1o is still in testing too and not marked stable.

[Toasterbirb]

Update: The latest release of mumble available on portage stopped compiling altogether on my system after an update

/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: src/mumble/CMakeFiles/mumble.dir/PluginManifest.cpp.o:(.data.rel.local.DW.ref._ZTIN4Poco3XML17SAXParseExceptionE[DW.ref._ZTIN4Poco3XML17SAXParseExceptionE]+0x0): undefined reference to `typeinfo for Poco::XML::SAXParseException'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
 * ERROR: media-sound/mumble-1.4.230::gentoo failed (compile phase):
 *   ninja -v -j8 -l0 failed
 *
 * Call stack:
 *     ebuild.sh, line  127:  Called src_compile
 *   environment, line 2157:  Called cmake_src_compile
 *   environment, line  850:  Called cmake_build
 *   environment, line  819:  Called eninja
 *   environment, line 1305:  Called die
 * The specific snippet of code:
 *       "$@" || die "${nonfatal_args[@]}" "${*} failed"
 *
 * If you need support, post the output of `emerge --info '=media-sound/mumble-1.4.230::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=media-sound/mumble-1.4.230::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/media-sound/mumble-1.4.230/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/media-sound/mumble-1.4.230/temp/environment'.
 * Working directory: '/var/tmp/portage/media-sound/mumble-1.4.230/work/mumble-1.4.230_build'
 * S: '/var/tmp/portage/media-sound/mumble-1.4.230/work/mumble-1.4.230.src'

>>> Failed to emerge media-sound/mumble-1.4.230, Log file:

>>>  '/var/tmp/portage/media-sound/mumble-1.4.230/temp/build.log'

 * Messages for package media-sound/mumble-1.4.230:

 * ERROR: media-sound/mumble-1.4.230::gentoo failed (compile phase):
 *   ninja -v -j8 -l0 failed
 *
 * Call stack:
 *     ebuild.sh, line  127:  Called src_compile
 *   environment, line 2157:  Called cmake_src_compile
 *   environment, line  850:  Called cmake_build
 *   environment, line  819:  Called eninja
 *   environment, line 1305:  Called die
 * The specific snippet of code:
 *       "$@" || die "${nonfatal_args[@]}" "${*} failed"

Are there binaries available for linux anywhere I could try for this specific version 1.4.230?

[ppfeufer]

@Toasterbirb :: This is a Gentoo issue, you might want to report it here » https://bugs.gentoo.org/

Also, check if it's already reported by any chance. (https://bugs.gentoo.org/buglist.cgi?quicksearch=mumble&list_id=6173459)

By the looks of it (the error message you get), you can try to re-emerge dev-util/ninja and then emerge mumble.

[lbratch]

I get the same issue when using Mumble 1.4.230 on Gentoo. The same crash when transmitting (after seconds to minutes), and the same sort of backtraces where I mostly (but not always) see a crash when inside OpenSSL.

Curiously I only get the crash when running through PipeWire (whether Mumble itself is using its ALSA backend via PipeWire, its PulseAudio backend via PipeWire, or its native PipeWire backend). When using Mumble's ALSA backend via pure ALSA, I do not get the crash.

I have fixed the issue by using the current master (4f50172c5c8bc7c425efb350377106d3e83a7e79) but with the Opus related commit a4fdc46a80d7da23ba02fa08c3361503b5cdc33c reverted as per #5302.

Edit: I should say that I /think/ it's fixed - I have tested transmitting in the current master for ~15 minutes without a crash which is significantly longer than it's lasted w/ 1.4.230 + PipeWire. This is using Mumble's ALSA backend via PipeWire. I will try to use it normally using Mumble's PipeWire backend tomorrow and report back if there are any issues.

[Toasterbirb]

I tried the latest master commit and used the cmake .. -Doverlay-xcompile=OFF -Dbundled-opus=OFF cmake command.

Mumble has yet to crash. Seems like the problem is fixed :tada:

[0xC0ncord]

I've also been seeing this crash in libcrypto.so in 1.4.230 on Gentoo but the current master at 414ab61b61b0d446eccba1c60dcad7c158a7c701 seems to have fixed it. I didn't need to alter any of the build flags or revert any commits.

Is there any indication as to what commit might've resolved this?