Draft: Migrate from SDL2 to SDL3
First, this PR contains some fixes that could be merged directly into dev.
Then, there are some refactors to make the transition easier, especially the handling of integer scrolling (https://github.com/Genymobile/scrcpy/issues/6156#issuecomment-3016836940) and the configuration of the color space at texture creation.
Finally, there's a "big" migration commit, followed by the changes for the build script and the GitHub Actions script.
Links:
- SDL3 migration docs: https://wiki.libsdl.org/SDL3/README-migration
- Relevant discussion about the new SDL3 audio API for scrcpy: https://github.com/libsdl-org/SDL/issues/13319
- Release binaries from GitHub Actions (download
scrcpy-release-sdl3): https://github.com/rom1v/scrcpy/actions/runs/16226120027
Tests and reviews welcome.
TODO:
- check if the "continuous resizing" workaround is still necessary, or if it can be removed (0b1e59186f1fb75d74f2929f636128958b067ab4)
- is it possible with SDL3 to keep the window aspect ratio while resizing (instead of adding black bars)? (#3460)
- other things?
I don't plan to merge this immediately, SDL3 is not available yet on many distros (not even in current Debian stable).
I'd like to implement hardware decoding first (#3800 | https://github.com/libsdl-org/SDL/issues/5405#issuecomment-1754956025 | https://github.com/libsdl-org/SDL/pull/8366), and if everything works as expected, release a new major version (scrcpy 4.0).
Quickly tried it on Windows (using the prebuilt binary) and it keeps printing Unexpected SDL audio behavior: too much data requested
┖[~\Downloads\scrcpy-win64-sdl3]> .\scrcpy.exe
scrcpy 3.3.1 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO: --> (usb) 73294bba device M2011K2C
C:\Users\simon\Downloads\scrcpy-win64-sdl3\scrcpy-server: ...file pushed, 0 skipped. 112.9 MB/s (90788 bytes in 0.001s)
[server] INFO: Device: [Xiaomi] Xiaomi M2011K2C (Android 13)
INFO: Renderer: direct3d11
INFO: Texture: 1440x3200
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
Enabling -Vverbose shows more [Audio] Buffering threshold exceeded messages
ERROR: Unexpected SDL audio behavior: too much data requested
DEBUG: [Audio] Buffering threshold exceeded, skipping 240 samples
ERROR: Unexpected SDL audio behavior: too much data requested
DEBUG: [Audio] Buffering threshold exceeded, skipping 480 samples
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
DEBUG: [Audio] Buffering threshold exceeded, skipping 480 samples
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
DEBUG: [Audio] Buffering threshold exceeded, skipping 240 samples
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
DEBUG: [Audio] Buffering threshold exceeded, skipping 480 samples
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
DEBUG: [Audio] Buffering threshold exceeded, skipping 480 samples
ERROR: Unexpected SDL audio behavior: too much data requested
ERROR: Unexpected SDL audio behavior: too much data requested
DEBUG: [Audio] Buffering threshold exceeded, skipping 480 samples
DEBUG: [Audio] Buffering threshold exceeded, skipping 960 samples
Thank you for your tests!
I assumed (on purpose, to detect early if that was false) that the additional_amount requested by SDL3 never exceeds the "audio output buffer" hint. It is true on my machine, but apparently not everywhere (I only tested on Linux).
How much data does it require:
diff --git app/src/audio_player.c app/src/audio_player.c
index e1a4134e7..6ca13a1e0 100644
--- app/src/audio_player.c
+++ app/src/audio_player.c
@@ -21,7 +21,8 @@ sc_audio_player_stream_callback(void *userdata, SDL_AudioStream *stream,
assert(len <= ap->aout_buffer_size);
if (len > ap->aout_buffer_size) {
// Just in case for release builds
- LOGE("Unexpected SDL audio behavior: too much data requested");
+ LOGE("Unexpected SDL audio behavior: too much data requested: %d",
+ additional_amount);
len = ap->aout_buffer_size;
}
?
Is it different with --audio-output-buffer=10 or --audio-output-buffer=20? (default is 5)
On SDL2, do you need to use --audio-output-buffer=xx to get audio working correctly?
How much data does it require:
additional_amount is always 3840, and audio sounds like a robot
Is it different with
--audio-output-buffer=10or--audio-output-buffer=20?
There is no error with either --audio-output-buffer=10 or --audio-output-buffer=20, additional_amount is still 3840, and audio is working
On SDL2, do you need to use --audio-output-buffer=xx to get audio working correctly?
No
There is no error with either
--audio-output-buffer=10or--audio-output-buffer=20, and audio is working
What is the value of additional_amount in these cases?
@icculus Sorry to ping you here, but do you know why setting SDL_HINT_AUDIO_DEVICE_SAMPLE_FRAMES to 240 (5 milliseconds of samples, 1920 bytes for 2 channels of 4 bytes each) can result in the callback being called with additional_amount = 3840 (instead of 1920) on Windows?
Apparently, on SDL2, the callback was correctly called with 1920 bytes. @yume-chan, this can be confirmed on scrcpy 3.3.1 with:
diff --git app/src/audio_player.c app/src/audio_player.c
index 9413c2ea8..07979b31b 100644
--- app/src/audio_player.c
+++ app/src/audio_player.c
@@ -11,6 +11,8 @@ static void SDLCALL
sc_audio_player_sdl_callback(void *userdata, uint8_t *stream, int len_int) {
struct sc_audio_player *ap = userdata;
+ LOGI("=== SDL2 audio callback: %d bytes (%d samples)", len_int, len_int / 8);
+
assert(len_int > 0);
size_t len = len_int;
Not awake yet, but I'll try the Windows build today and see what happens.
scrcpy with SDL3 can even be built on macOS Catalina with MacOSX_SDK11 using
SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk ./release/build_macos.sh x86_64
and it works splendidly, both with OpenGL and Metal. I have an older Android phone, so can't test audio.
diff --git a/app/deps/sdl.sh b/app/deps/sdl.sh
index ae79ae9c..a308ae9e 100755
--- a/app/deps/sdl.sh
+++ b/app/deps/sdl.sh
@@ -28,7 +28,7 @@ export CXXFLAGS="$CFLAGS"
if [[ -d "$DIRNAME" ]]
then
- echo "'$PWD/$HDIRNAME' already exists, not reconfigured"
+ echo "'$PWD/$DIRNAME' already exists, not reconfigured"
cd "$DIRNAME"
else
mkdir "$DIRNAME"
@@ -41,7 +41,7 @@ else
if [[ "$HOST" == linux ]]
then
conf+=(
- -DSDL_WAYLOAD=ON
+ -DSDL_WAYLAND=ON
-DSDL_X11=ON
)
fi
Also tested with SDL 3.2.18 and it works fine on macOS. Audio not tested.
diff --git a/app/deps/sdl.sh b/app/deps/sdl.sh
index ae79ae9c..970fcbfa 100755
--- a/app/deps/sdl.sh
+++ b/app/deps/sdl.sh
@@ -5,10 +5,10 @@ cd "$DEPS_DIR"
. common
process_args "$@"
-VERSION=3.2.16
+VERSION=3.2.18
FILENAME=SDL-$VERSION.tar.gz
PROJECT_DIR=SDL-release-$VERSION
-SHA256SUM=a1af401fb49b824ba721683862439e014498f79753602827e174f3613357aff0
+SHA256SUM=51539fa13e546bc50c632beed3f34257de2baa38a4c642048de56377903b4265
cd "$SOURCES_DIR"
What is the value of
additional_amountin these cases?
It's still 3840
diff --git a/app/src/audio_player.c b/app/src/audio_player.c
index e1a4134e..71fbef2a 100644
--- a/app/src/audio_player.c
+++ b/app/src/audio_player.c
@@ -16,6 +16,8 @@ sc_audio_player_stream_callback(void *userdata, SDL_AudioStream *stream,
struct sc_audio_player *ap = userdata;
if (additional_amount > 0) {
+ LOGE("additional_amount: %d", additional_amount);
+
size_t len = additional_amount;
assert(len <= ap->aout_buffer_size);
@@ -82,6 +84,9 @@ sc_audio_player_frame_sink_open(struct sc_frame_sink *sink,
int r = snprintf(str, sizeof(str), "%" PRIu16, (uint16_t) aout_samples);
assert(r >= 0 && (size_t) r < sizeof(str));
(void) r;
+
+ LOGE("setting SDL_HINT_AUDIO_DEVICE_SAMPLE_FRAMES to %s", str);
+
if (!SDL_SetHint(SDL_HINT_AUDIO_DEVICE_SAMPLE_FRAMES, str)) {
LOGE("Could not set audio output buffer");
sc_audio_regulator_destroy(&ap->audioreg);
> ./scrcpy --audio-output-buffer=10
scrcpy 3.3.1 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO: --> (usb) 73294bba device M2011K2C
D:\dev\yume-chan\scrcpy-devcontainer\scrcpy\server\build\o...file pushed, 0 skipped. 102.8 MB/s (91004 bytes in 0.001s)
[server] INFO: Device: [Xiaomi] Xiaomi M2011K2C (Android 13)
INFO: Renderer: direct3d11
ERROR: setting SDL_HINT_AUDIO_DEVICE_SAMPLE_FRAMES to 480
ERROR: additional_amount: 3840
ERROR: additional_amount: 3840
ERROR: additional_amount: 3840
ERROR: additional_amount: 3840
ERROR: additional_amount: 3840
ERROR: additional_amount: 3840
> ./scrcpy --audio-output-buffer=20
scrcpy 3.3.1 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO: --> (usb) 73294bba device M2011K2C
D:\dev\yume-chan\scrcpy-devcontainer\scrcpy\server\build\o...file pushed, 0 skipped. 107.7 MB/s (91004 bytes in 0.001s)
[server] INFO: Device: [Xiaomi] Xiaomi M2011K2C (Android 13)
INFO: Renderer: direct3d11
ERROR: setting SDL_HINT_AUDIO_DEVICE_SAMPLE_FRAMES to 960
ERROR: additional_amount: 3840
ERROR: additional_amount: 3840
ERROR: additional_amount: 3840
INFO: Texture: 1440x3200
ERROR: additional_amount: 3840
ERROR: additional_amount: 3840
this can be confirmed on scrcpy 3.3.1
> ./scrcpy
scrcpy 3.3.1 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO: --> (usb) 73294bba device M2011K2C
D:\dev\yume-chan\scrcpy-devcontainer\scrcpy\server\build\o... file pushed, 0 skipped. 99.5 MB/s (91004 bytes in 0.001s)
[server] INFO: Device: [Xiaomi] Xiaomi M2011K2C (Android 13)
INFO: Renderer: direct3d
INFO: Texture: 1440x3200
ERROR: === SDL2 audio callback: 1920 bytes (240 samples)
ERROR: === SDL2 audio callback: 1920 bytes (240 samples)
ERROR: === SDL2 audio callback: 1920 bytes (240 samples)
ERROR: === SDL2 audio callback: 1920 bytes (240 samples)
ERROR: === SDL2 audio callback: 1920 bytes (240 samples)
> ./scrcpy --audio-output-buffer=10
scrcpy 3.3.1 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO: --> (usb) 73294bba device M2011K2C
D:\dev\yume-chan\scrcpy-devcontainer\scrcpy\server\build\o...file pushed, 0 skipped. 104.3 MB/s (91004 bytes in 0.001s)
[server] INFO: Device: [Xiaomi] Xiaomi M2011K2C (Android 13)
INFO: Renderer: direct3d
INFO: Texture: 1440x3200
ERROR: === SDL2 audio callback: 3840 bytes (480 samples)
ERROR: === SDL2 audio callback: 3840 bytes (480 samples)
ERROR: === SDL2 audio callback: 3840 bytes (480 samples)
ERROR: === SDL2 audio callback: 3840 bytes (480 samples)
ERROR: === SDL2 audio callback: 3840 bytes (480 samples)
ERROR: === SDL2 audio callback: 3840 bytes (480 samples)
> ./scrcpy --audio-output-buffer=20
scrcpy 3.3.1 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO: --> (usb) 73294bba device M2011K2C
D:\dev\yume-chan\scrcpy-devcontainer\scrcpy\server\build\o...file pushed, 0 skipped. 103.7 MB/s (91004 bytes in 0.001s)
[server] INFO: Device: [Xiaomi] Xiaomi M2011K2C (Android 13)
INFO: Renderer: direct3d
INFO: Texture: 1440x3200
ERROR: === SDL2 audio callback: 7680 bytes (960 samples)
ERROR: === SDL2 audio callback: 7680 bytes (960 samples)
ERROR: === SDL2 audio callback: 7680 bytes (960 samples)
ERROR: === SDL2 audio callback: 7680 bytes (960 samples)
ERROR: === SDL2 audio callback: 7680 bytes (960 samples)
Numbers and punctuation now only work with --raw-key-events.
https://github.com/libsdl-org/SDL/blob/bc5c9a686ca01c7cffa45e223efcd0cb7ef1efb7/src/audio/wasapi/SDL_wasapi.c#L759-L760
SDL3 calls IAudioClient3::GetSharedModeEnginePeriod and clamps device->sample_frames. min_period_in_frames and max_period_in_frames are both 480, so additional_amount is always 3840. Not sure what it means, nor how does SDL2 works.
directsound audio driver doesn't work with the default --audio-output-buffer=5 neither, additional_amount is 1920 as expected, but audio played is broken:
log
> ./scrcpy --audio-output-buffer=5
scrcpy 3.3.1 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO: --> (usb) 73294bba device M2011K2C
DEBUG: Device serial: 73294bba
DEBUG: Using SCRCPY_SERVER_PATH: D:\dev\yume-chan\scrcpy-devcontainer\scrcpy\server\build\outputs\apk\release\server-release-unsigned.apk
D:\dev\yume-chan\scrcpy-devcontainer\scrcpy\server\build\o...file pushed, 0 skipped. 104.2 MB/s (91004 bytes in 0.001s)
[server] INFO: Device: [Xiaomi] Xiaomi M2011K2C (Android 13)
[server] DEBUG: Using audio encoder: 'c2.android.opus.encoder'
[server] DEBUG: Using video encoder: 'c2.qti.avc.encoder'
[server] DEBUG: Display: using SurfaceControl API
DEBUG: Server connected
DEBUG: Starting controller thread
DEBUG: Starting receiver thread
DEBUG: Using icon (portable): D:\dev\yume-chan\scrcpy-devcontainer\scrcpy\release\work\build-win64\dist\icon.png
INFO: Renderer: direct3d11
DEBUG: Trilinear filtering disabled (not an OpenGL renderer)
DEBUG: Demuxer 'video': starting thread
DEBUG: Demuxer 'audio': starting thread
INFO: Texture: 1440x3200
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
DEBUG: [Audio] Buffering threshold exceeded, skipping 240 samples
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
DEBUG: [Audio] Buffering threshold exceeded, skipping 480 samples
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
DEBUG: [Audio] Buffering threshold exceeded, skipping 480 samples
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
DEBUG: [Audio] Buffering threshold exceeded, skipping 480 samples
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
DEBUG: [Audio] Buffering threshold exceeded, skipping 480 samples
ERROR: additional_amount: 1920
ERROR: additional_amount: 1920
DEBUG: [Audio] Buffering threshold exceeded, skipping 240 samples
Numbers and punctuation now only work with
--raw-key-events.
Oh, SDL_EVENT_TEXT_INPUT is never received: https://github.com/Genymobile/scrcpy/blob/3615c75ad6074f088cddb0616410022ce7c0249b/app/src/input_manager.c#L1019
Reported here: https://github.com/libsdl-org/SDL/issues/13380
directsoundaudio driver doesn't work with the default--audio-output-buffer=5neither,additional_amountis 1920 as expected, but audio played is broken
Is it also the case on SDL2?
--audio-output-buffer was added because some audio outputs on Windows did not support 5ms, so if the behavior was the same on SDL2 (and respects the sample_frames value), it's ok I guess: https://github.com/Genymobile/scrcpy/issues/3793
Oh, SDL_EVENT_TEXT_INPUT is never received
Should be fixed on the new branch.
I added:
diff --git app/src/screen.c app/src/screen.c
index 27259e830..6ecb64167 100644
--- app/src/screen.c
+++ app/src/screen.c
@@ -394,6 +394,12 @@ sc_screen_init(struct sc_screen *screen,
goto error_destroy_fps_counter;
}
+ ok = SDL_StartTextInput(screen->window);
+ if (!ok) {
+ LOGE("Could not enable text input: %s", SDL_GetError());
+ goto error_destroy_window;
+ }
+
SDL_Surface *icon = scrcpy_icon_load();
if (icon) {
SDL_SetWindowIcon(screen->window, icon);
(cf https://github.com/libsdl-org/SDL/issues/13380)
As a consequence, it might open an IME on scrcpy start (https://github.com/libsdl-org/SDL/issues/13380#issuecomment-3083668746). If that's the case (to be tested), we need a workaround.
directsoundaudio driver doesn't work with the default--audio-output-buffer=5neither,additional_amountis 1920 as expected, but audio played is brokenIs it also the case on SDL2?
Yes, with SDL2, directsound driver, and --audio-output-buffer=5, audio playback is also broken. wasapi driver works.
check if the "continuous resizing" workaround is still necessary, or if it can be removed (https://github.com/Genymobile/scrcpy/commit/0b1e59186f1fb75d74f2929f636128958b067ab4)
It still doesn't work, video stream pauses when moving or resizing window.
I think https://github.com/libsdl-org/SDL/issues/1059#issuecomment-1802723797 only applies automatically when using main callbacks, otherwise the SC_EVENT_NEW_FRAME custom events are still not delivered (https://github.com/Genymobile/scrcpy/issues/3458#issuecomment-2768988367)
Thank you :+1:
Maybe it can make sense to use SDL main callbacks (to be tested). This will require to rework the way the scrcpy-server is started, but in the end in might simplify (or not).
@yume-chan
=== SDL2 audio callback: 1920 bytes (240 samples)
Could you please also add the timestamp of the call, to see if the callback is called twice in a row: https://github.com/libsdl-org/SDL/issues/13397
I don't plan to merge this immediately, SDL3 is not available yet on many distros (not even in current Debian stable).
Why not use a statically linked lib? Unless it's lots of megabytes in size but I'd still accept the tradeoff for better color accuracy.
Why not use a statically linked lib
This is one of the possibilities: https://github.com/Genymobile/scrcpy/blob/master/doc/linux.md#from-the-official-release
But I would like to avoid a dependency "too recent", which is not even packaged (Debian stable is my reference, if it's in stable, it's not "too recent").
Anyway, it will take some time to resolve the remaining issues with SDL3.
Hi,
I compiled the SDL3 branch in Termux, and it works well with the default settings. However, when using the --render=vulkan option, the colors appear inverted, although scrcpy remains usable.
In contrast, the precompiled build linked in the first post, which I ran on Windows, performs much worse. When launched with the --render=vulkan parameter, the screen displays only glitches, and the image is completely unreadable.
Downloads\scrcpy-release-sdl3\scrcpy-win64-sdl3> ./scrcpy --render=vulkan --no-audio
scrcpy 3.3.1 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO: --> (usb) ec8faaa3 device xxxxxxxxxxx
C:\Users\xxxxxxx\Downloads\scrcpy-release-sdl3\scrcpy-... file pushed, 0 skipped. 27.7 MB/s (90788 bytes in 0.003s)
[server] INFO: Device: [Xiaomi] Redmi xxxxxxxxxxxxxxxxxx (Android 13)
INFO: Renderer: vulkan
INFO: Texture: 1224x2712
If there’s anything I can do to help diagnose or resolve this issue, please let me know. I’m happy to provide logs or any other information needed.
For reference, on Windows 11 I have the latest graphics drivers installed for the Intel® Core™ Ultra 5 Processor 135U.
Thanks!
Looks like there is an alignment issue. Maybe because in 1224x2712, 1224 and 2712 are not multiple of 16. (see #55)
Try -m1024 or something else until you get both dimensions multiple of 16.
Can you reproduce with scrcpy + SDL2?
I can't reproduce this using SDL2 because the version built on github doesn't offer a Vulkan renderer.
PS C:\Users\xxxxxxxxxxxx\Downloads\scrcpy-release-sdl3\scrcpy-win64-sdl3> ./scrcpy --render=vulkan --no-audio -m1024
scrcpy 3.3.1 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO: --> (usb) ec8faaa3 device xxxxxxxxxxxx
C:\Users\xxxxxxxxxxxx\Downloads\scrcpy-release-sdl3\scrcpy-... file pushed, 0 skipped. 28.7 MB/s (90788 bytes in 0.003s)
[server] INFO: Device: [Xiaomi] Redmi xxxxxxxxxxxx (Android 13)
INFO: Renderer: vulkan
INFO: Texture: 464x1024
This parameter didn't help. The problem is the same. I tested several other renderers, and the ones I tested were: -direct3d12 -direct3d11 -opengl
The problem doesn't occur. The problem only occurs with the Vulkan renderer.
I also just tested the "direct3d" renderer and everything works fine there. The problem is only with the "vulkan" renderer.
Indeed, I can reproduce on Linux. Even with dimensions which are a multiple of 16 (or even 32).
Thank you for you report.
I created an issue here: https://github.com/libsdl-org/SDL/issues/13734
Thank you very much for your commitment and project. I have a question about SDL3. Does it bring any other benefits besides hardware acceleration of video decoding and DirectX 11, 12, Vulkan rendering?
I'm asking in the context of scrcpy
Paste doesn't work on macOS. Tried SDL 3.2.16 3.2.18 3.2.20.
SDL issue filed: https://github.com/libsdl-org/SDL/issues/13737
Copy works and sometimes Control-Shift-V works.
I updated the PR to not assume that the SDL SAMPLES_FRAMES hint is honored (refs https://github.com/libsdl-org/SDL/issues/13397). @yume-chan this should fix https://github.com/Genymobile/scrcpy/pull/6216#issuecomment-3072768615.
@adamponi The vulkan driver issue is fixed: https://github.com/libsdl-org/SDL/issues/13734
Does it bring any other benefits besides hardware acceleration of video decoding and DirectX 11, 12, Vulkan rendering?
Besides hardware acceleration (which requires some work), the main benefit is to use a version of SDL that will continue to evolve (not become abandoned/deprecated).