Migrate Android framework to SDL

[hwsmm]

This PR lets Android devices use SDL for windowing/input. osuTK usage got removed in the process of doing it. It also removes Android input handling code and replaces them with existing SDL components.

I did test using Vulkan with it and it kind of works, but it was just full of glitches (a lot of flashes and tearing). Veldrid backends (OpenGL/Vulkan) crash if they are brought up from background. I don't know well about graphics, anyway.. the only graphics backend that works well is Legacy OpenGL.

It contains non-C# components (C/Java) and source is available in my another branch, and SDL was built from commit https://github.com/libsdl-org/SDL/commit/41bf6b5a51d868061c767affbceea2d8735dcccd.

Java is needed because SDL Android heavily relies on JNI, and its binding is made with Xamarin.Android, and for C, SDL Android looks for the SDL main function from dynamic library symbol. Let me know if there's any better way. I'd be happy to make it better.

I didn't reuse the code, but brought an idea from https://github.com/0x0ade/SDL2Droid-CS, it helped me a lot!

Tested on my tablet/phone with Android 13.

You can use this osu! branch if you want to test.

[bdach]

Interesting that you got this to work (I tried what sounds to be the exact same approach, more or less, and it didn't work - input was broken, and so was debugging). Interested to see if this actually works.

The packaging 200% needs work though. We cannot be consuming random binaries from random sources. The build toolchain for our SDL2 fork (https://github.com/ppy/SDL2-CS) must be used for SDL binaries. Maybe the extra wrapper code must live there too (or else, in a separate repository), and if possible, the changes to the SDL Android project should be upstreamed.

[hwsmm]

I understand that reproducibility and integrity matter a lot in packaging, so I even considered excluding my binaries from this PR, but I thought it would cause some big inconvenience for testing, so I just went with having binaries included. I should have mentioned it.

I will look into that repo and open another PR there if I come up with an okay solution.

[Susko3]

SDL Android looks for the SDL main function from dynamic library symbol. Let me know if there's any better way. I'd be happy to make it better.

You could try using UnmanagedCallersOnlyAttribute for this. I think this will expose the function from the managed osu.Framework.Android.dll or SampleGame.Android.dll so hopefully SDL might be able to see it. I think that this requires compiling with NativeAOT.

With this, SDL2AndroidMainSetter should be completely unnecessary.

Example code from another project, here the native library expects a obs_module_load to be exposed on the DLL, this project is compiled with NativeAOT and the native library can see and call the function.

[bdach]

I think that this requires compiling with NativeAOT.

I'm not sure it will be relevant but the last time the words "AOT" and "android" were in close proximity things fizzled out very quickly due to being terminally broken. So I'm not holding my breath...

[hwsmm]

Above PR adds SDL2-CS-Android project to SDL2-CS. Changes in Java code are no longer needed, too. I'll commit changes that are needed from framework-side once it gets merged, since there isn't a NuGet package to depend on yet.

You can test it by removing osu.Framework.Android.SDL2 reference from osu.Framework.Android and add a reference to SDL2-CS-Android in osu.Framework.Android.

Edit: Tried AOT, but it doesn't seem to launch on my device...

[Susko3]

https://github.com/libsdl-org/SDL/commit/3a482ebae062cbc127e4e0faf9a07f69b54b7419 could be used to avoid the need for SDL2AndroidMainSetter. It's currently only available in SDL3, but we could backport it to SDL2.

[hwsmm]

I will just push my local framework as local SDL2-CS is needed to test at the moment anyway, and tidy up once things look okay.

With @Susko3's suggestion, I got it working and successfully removed SDL2AndroidMainSetter! You can test it by using SDL2-CS update-android-binaries branch.

[Joehuu]

On the framework apps (test and samplegame), I get a potential crash when rotating 90 degrees (osu! not affected as it can't rotate 90):

02-10 14:12:41.974 12820 12853 E AndroidRuntime: FATAL EXCEPTION: SDLThread
02-10 14:12:41.974 12820 12853 E AndroidRuntime: Process: osu.Framework.Tests.Android, PID: 12820
02-10 14:12:41.974 12820 12853 E AndroidRuntime: android.runtime.JavaProxyThrowable: [NUnit.Framework.AssertionException]: Stored Displays don't match actual displays: Stored displays:
02-10 14:12:41.974 12820 12853 E AndroidRuntime:   Name: 0, Bounds: {X=0,Y=0,Width=2400,Height=1080}, DisplayModes: 0, Index: 0
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 
02-10 14:12:41.974 12820 12853 E AndroidRuntime: Actual displays:
02-10 14:12:41.974 12820 12853 E AndroidRuntime:   Name: 0, Bounds: {X=0,Y=0,Width=1080,Height=2400}, DisplayModes: 0, Index: 0
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at osu.Framework.Logging.ThrowingTraceListener.Fail(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at System.Diagnostics.TraceInternal.Fail(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at System.Diagnostics.TraceInternal+TraceProvider.Fail(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at System.Diagnostics.Debug.Fail(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at System.Diagnostics.Debug.Assert(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at osu.Framework.Platform.SDL2Window.assertDisplaysMatchSDL(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at osu.Framework.Platform.SDL2Window.<handleWindowEvent>b__290_0(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at osu.Framework.Threading.ScheduledDelegate.InvokeTask(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at osu.Framework.Threading.ScheduledDelegate.RunTaskInternal(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at osu.Framework.Threading.Scheduler.Update(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at osu.Framework.Platform.SDL2Window.RunFrame(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at osu.Framework.Platform.SDL2Window.RunMainLoop(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at osu.Framework.Platform.SDL2Window.Run(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at osu.Framework.Platform.GameHost.Run(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at osu.Framework.Android.AndroidGameActivity.<CreateSDLMainRunnable>b__17_0(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at Java.Lang.Runnable.Run(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at Java.Lang.IRunnableInvoker.n_Run(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at Android.Runtime.JNINativeWrapper.Wrap_JniMarshal_PP_V(Unknown Source:0)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at mono.java.lang.Runnable.n_run(Native Method)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at mono.java.lang.Runnable.run(Runnable.java:31)
02-10 14:12:41.974 12820 12853 E AndroidRuntime: 	at java.lang.Thread.run(Thread.java:1012)

---

This may also resolve https://github.com/ppy/osu/issues/5236 for android, but there's a resizing bug which may be a blocker as it's really broken (on Pixel 6a with Android 14 beta if that changes anything):

https://github.com/ppy/osu-framework/assets/35318437/de145a8e-8590-48f9-a084-7b9d87e70ddb

[hwsmm]

I actually have run into that crash when I was writing this PR, but didn't fix it eventually since assertDisplaysMatchSDL is only present in Debug configuration. I fixed it in my last commit. https://github.com/ppy/osu-framework/blob/0e887087dad9c7c774d6f03c61403bf7d9fd7302/osu.Framework/Platform/SDL2Window_Windowing.cs#L309-L317

I can't reproduce the resizing bug in the video you posted, In the latest release (osuTK), my game doesn't get moved/resized at all, so I can't see what I am typing because keyboard hides bottom half, and in my build (SDL2), the game goes up (only showing bottom half), so I can see my chat fine.

My phone is Galaxy A51 on Android 13, so something may be different with keyboard handling, but I'm not really sure..

[Joehuu]

I can't seem to reproduce the resizing bug anymore. Probably incidentally fixed with the new commits.

The textboxes still have an off-by-one screen offset bug though:

• click/focus chat text box (no screen offset)
• click chat text box again (puts it into view)
• click settings text box (still offsets the screen where the chat text box used to be)

[smoogipoo]

This should now be able to consume https://www.nuget.org/packages/ppy.SDL3-CS.Android

[bdach]

I've added the package, and reverted the .sln changes since they seemed superfluous and .slns are a pain to review properly.

I did a smoke test, and while I would loooove to get this in, I'm hesitant. Despite the fact that this generally works better than when I tried this - on a brief smoke test on device 1 (yet to test device 2) things generally seem to work, aside from small issues like back button not working like it used to - the debuggability is zero. After a certain point in the game's launch sequence the activity just dies with debugger attached. I've never really loved the xamarin debugging experience - it's always sucked - but at least the debugger was available and now it's not.

I dunno how to feel about that, especially so that I'm the most likely to be on the hook when it comes to maintaining this in the future.

I can't reproduce @Joehuu's reported issue either, for whatever that's worth.

[bdach]

Aaaaand on test device number 2 this just dies on startup:

2024-04-19 12:21:51.912 13687-13706 AndroidRuntime          osu.Framework.Tests.Android          E  FATAL EXCEPTION: SDLThread
                                                                                                    Process: osu.Framework.Tests.Android, PID: 13687
                                                                                                    android.runtime.JavaProxyThrowable: [System.TypeLoadException]: Could not resolve type with token 010000a2 from typeref (expected class 'SDL.Utf8String' in assembly 'SDL3-CS, Version=2024.418.1.0, Culture=neutral, PublicKeyToken=null')
                                                                                                    	at Java.Lang.Runnable.Run(Unknown Source)
                                                                                                    	at Java.Lang.IRunnableInvoker.n_Run(Unknown Source)
                                                                                                    	at Android.Runtime.JNINativeWrapper.Wrap_JniMarshal_PP_V(Unknown Source)
                                                                                                    	at mono.java.lang.Runnable.n_run(Native Method)
                                                                                                    	at mono.java.lang.Runnable.run(Runnable.java:31)
                                                                                                    	at java.lang.Thread.run(Thread.java:762)

I dunno. I was hoping for easy wins here but this and the lack of debugger is probably going to take weeks to months to diagnose and fix if one wanted to.

[bdach]

@smoogipoo Not sure why you deleted comment (or not? I saw one but now it's not there) but full nuke of bin/ seems to have fixed the hard crash and the broken debugger? God I love this tooling.

Maybe we can move forward with this then. I'm not really inclined to do some proper test sweep on this, rather than just get it in and see how users respond because we're not gonna be able to gauge anything concretely otherwise. Unless @Susko3 wants to have a go I guess.

[smoogipoo]

Not sure why you deleted comment

Hmm, I deleted it because the exception says in assembly 'SDL3-CS, Version=2024.418.1.0, which is the latest version. Bizzare.

[Joehuu]

I can't reproduce @Joehuu's reported issue either, for whatever that's worth.

Tested again on three devices of varying versions: Google Pixel 6a (Android 14), Moto G Play 2021 (Android 11), and Moto G5 Plus (Android 8) and all exhibit the same problem. Here's a video of framework showing the same thing:

https://github.com/ppy/osu-framework/assets/35318437/a5810b41-edef-4b8b-a976-812287f67522

[hwsmm]

I applied most of the review suggestions, but there seem to be some remaining issues (keyboard screen offsets and multi-touch assert fail for now?), but I currently don't have much time to actually investigate into them.. I should be more free after a week, so I'll look into them then.

Though if anyone suggests fixes for the forementioned issues in the meantime, I'll try applying them (or just commit them yourself if you have perms?).

[Susko3]

The main problem with the keyboard is that we're calling SDL_StartTextInput before SDL_SetTextInputRect. This leads to the keyboard using the stale text input rect value.

From SDL_SetTextInputRect documentation:

To start text input in a given location, this function is intended to be called before SDL_StartTextInput, although some platforms support moving the rectangle even while text input (and a composition) is active.

I'll try to figure something out.

[Susko3]

I've reported https://github.com/ppy/osu-framework/pull/6105#pullrequestreview-2011866789 upstream in https://github.com/libsdl-org/SDL/issues/9591. This issue is not unique to android and can fail on Windows too.

[bdach]

@Susko3 is this fine to move forward after all of the other PRs?

[Susko3]

Yep this is good to go, just needs merge master + removal of no longer necessary workarounds:

1. Revert SDL3Window_Windowing.cs
2. AndroidGameWindow.Create() workarounds

Also the AndroidGameHost shouldn't be disposed.

[bdach]

Revert SDL3Window_Windowing.cs

All of it? What about this thing? I don't see it being moved anywhere?

[Susko3]

What about this thing? I don't see it being moved anywhere?

You're correct, that has to stay, I must have confused it with the dropped events that were fixed with https://github.com/ppy/osu-framework/pull/6259.

[bdach]

Also the AndroidGameHost shouldn't be disposed.

Have removed this in https://github.com/ppy/osu-framework/pull/6105/commits/dad23c0c65ae9448bfaceec9fdeb6b7ba8405059.

2. AndroidGameWindow.Create() workarounds

I can't seem to be able to remove these. Doing so results in the window never becoming focused and as such touch input doesn't work.

I'd really like to move this along now so I think this can be fixed as a follow-up.

[Susko3]

just need to remove the osuTK.Android reference as it's no longer needed.

Or you can merge right now, and we do this later in osuTK cleanup.