[MacOS] Game Crashing randomly

[VasilisThePikachu]

trafficstars

Description

Seen people report this on #help with the client's last last words in log being:

libc++abi: terminating

Now that im using a macbook i managed to expirience this crash and even grab the "stack trace" macos drops

Reproduction

1. Play on MacOS
2. Wait
3. Lol

Additional context

Crash dump macos generates automagicly macos m1 crash.txt

Edit: This seems to be an error with the built in openal library on macos, fixing will require #4997

[VasilisThePikachu]

Maybe should be moved to RT now that I think about it

[VasilisThePikachu]

Curiously enough... Either I was extremely lucky or this only happens on wizden servers? While I was admining on a fork I managed to stay for the entire round.

[VasilisThePikachu]

Seems to be a macos issue in general after talking back with another person. Who had a macbook pro intel instead.

[AverageNotDoingAnythingEnjoyer]

It's not connected to specific server, I think. In my experience, sometimes the game can crash 3 times in 10 minutes, sometimes it can avoid crashes for multiple hours or even until the user closes it manually, all on the same server. These occurred to me on two servers I played on, maybe it's the problem on all servers, but I haven't played on other ones in a while, so idk

[VasilisThePikachu]

Potentially found: Looking a bit at the crash log (and from someone adding their own OpenAL library), it seems to be the OpenAL library provided in MacOS to be causing issues. Which basically makes this an extension of #4997

[perryprog]

Unfortunately since updating to macOS Sequoia I've actually qualitatively noticed an increase in frequency in these crashes. (I'm often crashing every 5–30 minutes now!) The normal EXC_CRASH (SIGABRT) is still happening (see linked file in original post) but I'm now additionally getting a new crash that I don't remember seeing, though it is still OpenAL related. Here it's a EXC_BAD_INSTRUCTION (SIGILL), with the relevant part of the stack trace being:

Thread 11 Crashed:: .NET TP Worker
0   <translation info unavailable>	       0x10a237d10 ???
1   caulk                         	    0x7ff811937a3a caulk::alloc::bitmap_allocator<caulk::alloc::embed_block_memory, 16368ul, 16ul, 6ul>::deallocate(caulk::alloc::block, unsigned long) + 116
2   caulk                         	    0x7ff81193cfc6 caulk::alloc::tiered_allocator<caulk::alloc::size_range_tier<0ul, 1008ul, caulk::alloc::tree_allocator<caulk::alloc::chunk_allocator<caulk::alloc::page_allocator, caulk::alloc::bitmap_allocator, caulk::alloc::embed_block_memory, 16384ul, 16ul, 6ul>>>, caulk::alloc::size_range_tier<1009ul, 256000ul, caulk::alloc::guarded_edges_allocator<caulk::alloc::consolidating_free_map<caulk::alloc::page_allocator, 10485760ul>, 4ul>>, caulk::alloc::tracking_allocator<caulk::alloc::page_allocator>>::deallocate(caulk::alloc::block, unsigned long) + 120
3   AudioDSP                      	       0x1c655114b 0x1c5dc7000 + 7905611
4   AudioDSP                      	       0x1c654bded 0x1c5dc7000 + 7884269
5   AudioDSP                      	       0x1c654727c 0x1c5dc7000 + 7864956
6   AudioToolboxCore              	    0x7ff80865b728 AudioUnitSetProperty + 501
7   OpenAL                        	    0x7ffb0a2335ea OALSource::AddNotifyAndRenderProcs() + 80
8   OpenAL                        	    0x7ffb0a233355 OALSource::Play() + 1135
9   OpenAL                        	    0x7ffb0a228911 alSourcePlay + 26
10  ???                           	       0x119b055c2 ???
11  ???                           	       0x1199f6578 ???
... [snip]

As opposed to the relevant stack trace for the crash mentioned in the original post which looks more like:

Thread 23 Crashed:: .NET TP Worker
0   ???                           	    0x7ff8963eaa84 ???
1   libsystem_kernel.dylib        	    0x7ff80621cb52 __pthread_kill + 10
2   libsystem_pthread.dylib       	    0x7ff806256f85 pthread_kill + 262
3   libsystem_c.dylib             	    0x7ff806177bb9 __abort + 145
4   libsystem_c.dylib             	    0x7ff806177b28 abort + 141
5   libc++abi.dylib               	    0x7ff80620e163 abort_message + 258
6   libc++abi.dylib               	    0x7ff8061feff3 demangling_terminate_handler() + 47
7   libc++abi.dylib               	    0x7ff80620d53b std::__terminate(void (*)()) + 6
8   libc++abi.dylib               	    0x7ff80620d4e0 std::terminate() + 32
9   AudioDSP                      	       0x1c442814b 0x1c3c9e000 + 7905611
10  AudioDSP                      	       0x1c4422ded 0x1c3c9e000 + 7884269
11  AudioDSP                      	       0x1c441e27c 0x1c3c9e000 + 7864956
12  AudioToolboxCore              	    0x7ff80865b728 AudioUnitSetProperty + 501
13  OpenAL                        	    0x7ffb0a2335ea OALSource::AddNotifyAndRenderProcs() + 80
14  OpenAL                        	    0x7ffb0a233355 OALSource::Play() + 1135
15  OpenAL                        	    0x7ffb0a228911 alSourcePlay + 26
16  ???                           	       0x118048f72 ???
17  ???                           	       0x117cf9bd6 ???
18  ???                           	       0x117d0aafb ???
19  ???                           	       0x110983e94 ???
... [snip]

I had also tried to debug this a bit more previously (before I noticed this new form of crash) but not to much luck. I did try updating OpenAL from [email protected] to [email protected] but unfortunately this did not fix it.

I'm going to poke around a bit more and see what I can do, but I am very much a not-C# person so my knowledge is limited. I do however have a working dev environment set-up for SS14 and Robust so if anyone needs me to test anything I am very happy to! (CC @PJB3005, just in case.)

Finally, if anyone can get a reliable way to force the crash, that would be nice... currently my steps to reproduce are just join a local server and practice my tiding skills for anywhere from 10 to 30 minutes. It might happen more frequently when another client is logged in but that's just qualitative.

Full crash log of an example "bad instruction" crash: SS14.Loader-2024-10-14-011036.txt.

[perryprog]

And as a quick update:

• I haven't been able to get Rider to actually break on one of these signal interrupts.
• Registering a handler viaPosixSignalRegistration for signals 4 (SIGILL) and 6 (SIGABRT) doesn't seem to catch anything, but I could be using it wrong.
• The problematic call is definitively the SourcePlay call here, as opposed to the one in BufferedAudioSource (which AFAIK just used for MIDI anyway) https://github.com/space-wizards/RobustToolbox/blob/ba7d1452c1abcf0a2d83266d828d503d31d33c35/Robust.Client/Audio/Sources/BaseAudioSource.cs#L89-L96
• I cannot find any clear patterns for what conditions in playing a source actually causes an issue. (I've checked source state, source type, filters...) It'd be really great if anyone has ideas on what this may be, as that'd let us at least put in a workaround.

If this is indeed an issue with the macOS distributed (and deprecated!) OpenAL, then that could be problematic. We would really want to use CoreAudio but presumably isn't that something OpenTK should be using? I have to admit I'm more or less talking out of my butt here, as I'm not at all familiar with the OpenTK ecosystem or OpenAL. Maybe the best thing we can do is figure out what exact edge case is actually being hit here and just try to bodge in a fix for it on Robust Toolbox's side.

I've currently had good success at reproduction by just spawning in a bunch of syndicate footsoldiers and letting them go ham on a bunch of spawned in ERT ghost roles, which seems to be a good enough stress test that it'll usually cause the crash within about ten seconds.

[DoguhanOzgurAkca]

I had fixed the issue a bit ago by simply adding openAL dll into the files. Then after a few days, I started getting weird crashes, and now i get more crashes then ever. It has become extremely annoying.

[HelpTheSteak]

I thought it was just me all this time. Glad to know it ls just a global mac issue

[DoguhanOzgurAkca]

I’ve found a potential workaround for the issues some of us have been experiencing. After chatting with someone who’s been running the game on macOS 13.x without problems, I decided to test it out myself by dual-booting Monterey (12.x) alongside my current setup.

After playing about ten rounds of RMC 14, I only encountered two crashes, neither of which seemed related to the OpenAL issue. Interestingly, reconnecting to the game has improved significantly—it now takes less than a minute, whereas it used to take several.

If you’re interested in playing the game without the persistent issues, I recommend considering dual-booting your Mac. Keep in mind that it does require around 30 GB of space (with just Steam installed).

One last note: I couldn’t get MIDI working on the latest macOS, but it works perfectly on Monterey.

If anyone needs help setting this up, feel free to reach out on Discord!

[perryprog]

One last note: I couldn’t get MIDI working on the latest macOS, but it works perfectly on Monterey.

I've been able to have MIDI work on macOS 15 fine as long as you install an x86 version of fluidsynth that then runs under Rosetta. This is most easily done via an x86 installation of homebrew, which will then manage grabbing x86 versions of all of fluidsynth's dependencies and so on.

[DoguhanOzgurAkca]

One last note: I couldn’t get MIDI working on the latest macOS, but it works perfectly on Monterey.

I've been able to have MIDI work on macOS 15 fine as long as you install an x86 version of fluidsynth that then runs under Rosetta. This is most easily done via an x86 installation of homebrew, which will then manage grabbing x86 versions of all of fluidsynth's dependencies and so on.

exactly what I did! didn't work at 15, but worked at 12.

[DoguhanOzgurAkca]

To give a slightly more report since I been playing for quite a bit these days.

I started getting a little bit of crashes in monterey, so I jumped a bit through versions and now I am using Ventura. Crashing issue is COMPLETELY fixed... well not completely, I still get crashes from time to time, but its like, one per day or something at worst.

I can say one thing, since the crashing issue is fixed, I can play for extended amounts of times, and now the sound bug is the biggest problem. Constantly relogging made it so that it was not bad previously, now it haunts me.

[alterae]

i am on the latest macOS build and still experience quite a lot of crashes. subjectively, they seem to occur most when the chat sidebar is really busy, but that could very well be sampling bias

[DoguhanOzgurAkca]

i am on the latest macOS build and still experience quite a lot of crashes. subjectively, they seem to occur most when the chat sidebar is really busy, but that could very well be sampling bias

thats the problem. Current version is problematic, I am guessing it crept up somewhere around version 14.

Dual boot it, go to 13, play the game there. Works well. Can help over discord if you add me, I am in both RMC and SS14 servers with the name cure elim

[Quantum-cross]

My partner is experiencing this during normal games on her mac, so I installed Rider to try and debug it... I can't seem to recreate this at all, spawned in tons of syndie footsoldiers to make tons of sounds and I still can't make it happen.

[perryprog]

An interesting update: for a period of about two, close to maybe three months, I had a blissful period that was, inexplicably, free of these crashes. I have no idea why. There are still occasional, once or twice a day instances, where I'll crash, but it'll be with a slightly different backtrace in the relevant thread. Unfortunately, as of today, the common crash described here has returned with a vengeance and I cannot figure out why. It seems more inconsistent when I build locally compared to playing online, though I was able to get it to occur on a local build once but then not again. (Bisecting did no good due to this. I should also note that this is on Delta-V.)

However: Given the latest info in #4997, it might be the case that this issue can be resolved by properly getting macos-arm64 builds rolling. It'd be really, really great if this could happen!

[vidmaj]

I've been playing on MacOS Sonoma 14.5 for a while now, it continuously randomly crashes as described in this thread but if I am persistent on playing long enough - at some point the game will lose sound and it will become a crash-free experience. Not a dev here but seems like it could be related to sound - perhaps there is a quick workaround type of thing to disable sound altogether? I'd rather play without sound and crashes than the opposite.

[ThatOneGoblin25]

Doesn't seem like #5768 has fixed this issue? Still crashing at random when I load up ss14 on my MacBook.

[PJB3005]

Doesn't seem like #5768 has fixed this issue? Still crashing at random when I load up ss14 on my MacBook.

The launcher hasn't been updated yet. It's being worked on.

[PJB3005]

Presumably fixed by move to OpenAL Soft on latest launcher & engine.