Crash on iOS due to using GL when backgrounded (gpus_ReturnNotPermittedKillClient)

[Tw1ddle]

trafficstars

This issue is causing crashes for a few percent of iOS sessions, and happens across all iOS versions and devices. I am using the haxelib openfl 8.8.0, lime 7.2.1 and hxcpp 4.0.8.

It affects both the IMGSGX543GLDriver and AGXGLDriver so it doesn't appear to be a driver bug.

The crashdump stack ends at gpus_ReturnNotPermittedKillClient, which is the app getting killed for doing GL work after the app has been backgrounded, see: https://developer.apple.com/library/content/qa/qa1766/_index.html

This is the stack trace:

Thread 0 name: Thread 0 Crashed: 0 libGPUSupportMercury.dylib 0x00000001de8f1fe4 gpus_ReturnNotPermittedKillClient + 12 (gpui_client_io.c:78) 1 AGXGLDriver 0x00000001e2f21ed8 glrKillClient + 468 (agxu_kill_client.cpp:175) 2 libGPUSupportMercury.dylib 0x00000001de8f2fac gpusSubmitDataBuffers + 176 (gpui_context.c:0) 3 AGXGLDriver 0x00000001e2f23404 SubmitPackets(AGXContextRec*) + 292 (agxu_gl_command.cpp:93) 4 GLEngine 0x00000001e3fcd234 gliPresentViewES_Exec + 184 (gli_drawable_es.c:526) 5 OpenGLES 0x00000001c5c44aa4 -[EAGLContext presentRenderbuffer:] + 80 (eagl_context.m:594) 6 theapp 0x0000000100d910d4 -[SDL_uikitopenglview swapBuffers] + 320 7 theapp 0x0000000100d90264 UIKit_GL_SwapWindow + 84 8 theapp 0x0000000101205334 lime::_internal::backend::native::NativeWindow_obj::contextFlip() + 100 9 theapp 0x000000010156a798 lime::_internal::backend::native::NativeApplication_obj::handleRenderEvent() + 452 10 theapp 0x000000010156a7e0 lime::_internal::backend::native::__NativeApplication_objhandleRenderEvent(hx::Object*) + 20 11 theapp 0x000000010160a5dc val_call0 + 48 12 theapp 0x0000000100b293f8 lime::ValuePointer::Call() + 56 13 theapp 0x0000000100b1ce38 lime::SDLApplication::Update() + 52 14 QuartzCore 0x00000001c6dc3f90 CA::Display::DisplayLink::dispatch_items(unsigned long long, unsigned long long, unsigned long long) + 636 (CADisplay.mm:2349) 15 IOKit 0x00000001c2c984b0 IODispatchCalloutFromCFMessage + 488 (IOKitLib.c:1216) 16 CoreFoundation 0x00000001c29a7a8c __CFMachPortPerform + 188 (CFMachPort.c:522) 17 CoreFoundation 0x00000001c29ce690 CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION + 56 (CFRunLoop.c:1996) 18 CoreFoundation 0x00000001c29cdddc __CFRunLoopDoSource1 + 440 (CFRunLoop.c:2133) 19 CoreFoundation 0x00000001c29c8c00 __CFRunLoopRun + 2096 (CFRunLoop.c:3152) 20 CoreFoundation 0x00000001c29c80b0 CFRunLoopRunSpecific + 436 (CFRunLoop.c:3247) 21 GraphicsServices 0x00000001c4bc879c GSEventRunModal + 104 (GSEvent.c:2245) 22 UIKitCore 0x00000001eeb5a978 UIApplicationMain + 212 (UIApplication.m:4353)

Since this crash apparently happens when/after the app is backgrounded and in the main thread as Lime processes events from SDL, I'll walk through the sequence of events in the code below.

When the app is about to get backgrounded, SDL receives and handles the applicationWillResignActive lifecycle event:

https://github.com/spurious/SDL-mirror/blob/6c473135a31f79dfa4845b42e7c9d17f8d983b43/src/video/uikit/SDL_uikitappdelegate.m#L475

SDL tells the app about this by sending SDL_WINDOWEVENT_FOCUS_LOST, SDL_WINDOWEVENT_MINIMIZED window events and a SDL_APP_WILLENTERBACKGROUND app event:

https://github.com/SDL-mirror/SDL/blob/22ac8341d72d680739c2d55db194f465c12f98b7/src/video/SDL_video.c#L4039

https://github.com/SDL-mirror/SDL/blob/e85562f2ac6bf8e37f778ad5343e4e8f0a9abbd5/src/events/SDL_events.c#L965

Lime receives these events and handles them in a big switch statement:

https://github.com/openfl/lime/blob/290c2912a3d62d6b22d4fc3fb15cf3e764587433/project/src/backend/sdl/SDLApplication.cpp#L117-L121

Lime sets a boolean flag inBackground to true when the SDL_APP_WILLENTERBACKGROUND event is handled:

https://github.com/openfl/lime/blob/290c2912a3d62d6b22d4fc3fb15cf3e764587433/project/src/backend/sdl/SDLApplication.cpp#L145-L151

This inBackground flag is being used as a guard before calls to RenderEvent::Dispatch (&renderEvent);.

The only spot I notice missing this inBackground guard is the SDL_RENDER_DEVICE_RESET event, but I think these events should only get sent on Android or D3D11 targets anyway:

https://github.com/openfl/lime/blob/290c2912a3d62d6b22d4fc3fb15cf3e764587433/project/src/backend/sdl/SDLApplication.cpp#L232-L241

The rendering-related events are handled by the Lime native application class. The event that causes the crash appears to be a RENDER event, because the code that crashes in the stack trace listed above calls contextFlip:

https://github.com/openfl/lime/blob/b9bdf3479bafd6dfbc557e97dab4e4b5fcc0db9a/src/lime/_internal/backend/native/NativeApplication.hx#L364-L376

I can't see how this code above could be handling render events after the app is backgrounded, and haven't reproduced the issue myself yet, but it's crashing for users in the wild.

Note that I have not tried lime 7.5.0 yet due to a couple of different issues in SDL - but I can't find changes in recent versions of lime or SDL that could have fixed this.