openal-soft
openal-soft copied to clipboard
kcat
Oboe backend
With apologies if this has been discussed before, but I feel that a backend using Oboe sound system (https://github.com/google/oboe) is needed.
For some reason the existing OpenSL backend, doesn't play nice with android (no sound at all on some devices, extremely high audio latency on others, hearing loud "clicks", and probably other issues I've not encountered in my testings) making it pretty unusable.
The only feasible way to use the library on android I found is to use "android" backend, which uses the java AudioTrack class (via JNI). Unfortunately this backend was done for version 1.13 and, to my knowledge, never updated to support newer versions.
If Oboe is a normal/easy-to-get part of the Android ecosystem, it does seem to be a nice way to deal with some of the problems the OpenSL backend is experiencing. Luckily I also recently updated OpenAL Soft to use C++14, so that requirement is met too.
Though I'm not really sure how it's supposed to be used. It doesn't seem as though you build and install it (like you would e.g. PortAudio on Linux or Windows). The Getting Started guide says prebuilt libs/headers are available as a prefab package via Google Maven, which you can get using a preview version of Android Studio, but OpenAL Soft doesn't use that as a plain CMake project.
The other alternative mentioned is a submodule, which are never fun to use. And seems a bit overkill since it's not needed outside of Android builds. I suppose I could instead have the user set OBOE_DIR to the source directory if they want to build with Oboe support. Seems less than ideal, though.
Maybe I am missing something here, but I assumed the backend would just use "#include <oboe/Oboe.h>" and expect the user to setup their build environment in a way that the library is visible to the compiler.
Maybe, I'm not too familiar with Android development. From what little I do with it, I extract the NDK, and when building a CMake project (like Oboe) I set CMAKE_TOOLCHAIN_FILE to {NDK_ROOT}/build/cmake/android.toolchain.cmake, and it builds cross-compiled. Running make install tries to install it in /usr/local, which doesn't seem right, and there's no other clear place to install it to. Perhaps there's a better way to build/install projects for Android so they're automatically available to other projects.
I am not an expert in android development in any way, but I don't think it's possible to make a library visible to the toolchain as a whole. Maybe there's a way to manually copy the headers/binaries to NDK_ROOT/sysroot/usr/(local?) or something like that, but I don't think it's officially supported. Their own guide (https://developer.android.com/ndk/guides/prebuilts) assume the library binaries are copied in the project you are building. Isn't that enough though? I have to confess I have no idea what the official method of building openal-soft library for android is - in my tests I just create a shared library via self-made cmake script so linking OBOE in won't be a problem. If the official method is to use ndk-build, or standalone generated toolchain things may be different, but I can't imagine it to be too hard - probably just a few -I/-L/-l switches.
Their own guide (https://developer.android.com/ndk/guides/prebuilts) assume the library binaries are copied in the project you are building. Isn't that enough though? I have to confess I have no idea what the official method of building openal-soft library for android is - in my tests I just create a shared library via self-made cmake script so linking OBOE in won't be a problem.
For OpenAL Soft, I imagine most users either build it standalone using the NDK's cmake toolchain and place the binaries somewhere other projects can see them, or copy OpenAL Soft's source into a subdirectory of their project and add add_subdirectory(...) to their CMakeLists.txt.
For OpenAL Soft to use Oboe, though, having a copy in a subdirectory seems somewhat impractical since it's an optional dependency for one target system. That's probably fine for a bigger "end product" project, where the size of the dependency is minuscule compared to the whole, but for a smaller cross-platform library it seems rather cumbersome.
I can of course have the user supply the source if they want it and OpenAL Soft can add_subdirectory(...) on it, and/or use it if it finds it in the compiler's paths, I just don't know if that's an acceptable solution they'd be happy with. Or if there's a better method I'm not aware of.
Commit 0390828de06844514cde812e0475c6e1d338aad3 adds an initial attempt at an Oboe backend for playback. If the headers and prebuilt lib aren't visible to the compiler, you need to pass -DOBOE_SOURCE=... to cmake, where ... is the path to the Oboe source root directory, and it will be built with OpenAL Soft. I can't confirm that it actually works yet, but it does build.
~~Oddly, enabling this has caused Travis CI to start failing the Android build, even though I can watch it successfully build. I'm not sure what's going on with that.~~ Nevermind that, was a typo in the script.
Wow, that was fast!
I played around with it a bit, and from what little testing I've done, it seems to be working great. The latency seems considerably lower than the openes backend, and I didn't hear any cracks/glitches.
Unfortunately the only testing I've done was on an emulator since I don't have any android device around, and the stupid quarantine makes it somewhat hard to get one.
With that being said, I am extremely impressed by the results, and the time it took for it to happen, awesome work!
The latency seems considerably lower than the openes backend
Part of that is likely because it's just taking the default buffer metrics for low-latency output, and ignoring OpenAL Soft's buffer metrics. But that also means there's less protection against underruns if the device has trouble keeping up because it doesn't have the usual buffer size.
I updated the code a bit to trace what the configured stream is like, and it'd be interesting to see a log of what it gets.
It should be written to the Android system(?) log, though I don't know how to get that with an emulator (it's been a while and I don't remember how to see it with a real system/phone either). Alternatively, you can set the ALSOFT_LOGLEVEL env var to 3 when you run the app and it'll be written to stderr.
Here's the output I got from adb logcat | grep "AL lib", not sure what to make of it, but hopefully it would be helpful to you. The log was taken from a x86 emulator.
D/openal: AL lib: Initializing library v1.20.1-253773b3 master D/openal: AL lib: Supported backends: oboe, null D/openal: AL lib: Loading config /etc/openal/alsoft.conf... D/openal: AL lib: Loading config /etc/xdg/alsoft.conf... D/openal: AL lib: Got binary: /system/bin, app_process32 D/openal: AL lib: Loading config /system/bin/alsoft.conf... D/openal: AL lib: Key disable-cpu-exts not found D/openal: AL lib: Extensions: -none- D/openal: AL lib: Key rt-prio not found D/openal: AL lib: Key resampler not found D/openal: AL lib: Key trap-al-error not found D/openal: AL lib: Key trap-alc-error not found D/openal: AL lib: Key reverb/boost not found D/openal: AL lib: Key drivers not found D/openal: AL lib: Initialized backend "oboe" D/openal: AL lib: Added "oboe" for playback D/openal: AL lib: Initialized backend "null" W/openal: AL lib: No capture backend available! D/openal: AL lib: Key excludefx not found D/openal: AL lib: Key default-reverb not found D/openal: AL lib: Key channels not found D/openal: AL lib: Key sample-type not found D/openal: AL lib: Key frequency not found D/openal: AL lib: Key period_size not found D/openal: AL lib: Key periods not found D/openal: AL lib: Key sources not found D/openal: AL lib: Key slots not found D/openal: AL lib: Key sends not found D/openal: AL lib: Key ambi-format not found D/openal: AL lib: Created device 0x93435f10, "" D/openal: AL lib: ALC_FREQUENCY = 44100 D/openal: AL lib: Key frequency not found D/openal: AL lib: Key period_size not found D/openal: AL lib: Key periods not found D/openal: AL lib: Key sources not found D/openal: AL lib: Key sends not found D/openal: AL lib: Key hrtf not found D/openal: AL lib: Pre-reset: Stereo, Float, *44100hz, 882 / 2646 buffer D/openal: AL lib: Got stream with properties: StreamID: 0xa5519c00 DeviceId: 0 Direction: Output API type: OpenSLES BufferCapacity: 384 BufferSize: 384 FramesPerBurst: 192 FramesPerCallback: 192 SampleRate: 44100 ChannelCount: 2 Format: Float SharingMode: Shared PerformanceMode: None CurrentState: Open XRunCount: 0 FramesRead: 0 FramesWritten: 0 D/openal: AL lib: Post-reset: Stereo, Float, 44100hz, 441 / 882 buffer D/openal: AL lib: Key stereo-mode not found D/openal: AL lib: Key cf_level not found D/openal: AL lib: Key stereo-encoding not found D/openal: AL lib: Stereo rendering D/openal: AL lib: Channel config, Main: 3, Real: 2 D/openal: AL lib: Allocating 5 channels, 20480 bytes D/openal: AL lib: Enabling first-order ambisonic decoder D/openal: AL lib: Max sources: 256 (255 + 1), effect slots: 64, sends: 2 D/openal: AL lib: Front stablizer disabled D/openal: AL lib: Key dither not found D/openal: AL lib: Key dither-depth not found D/openal: AL lib: Dithering disabled D/openal: AL lib: Key output-limiter not found D/openal: AL lib: Output limiter disabled D/openal: AL lib: Fixed device latency: 0ns D/openal: AL lib: Increasing allocated voices to 256 D/openal: AL lib: Key volume-adjust not found D/openal: AL lib: Created context 0x9340a1d0 W/openal: AL lib: Querying error state on null context (implicitly 0xa004)
And one more log, taken from a really crappy tablet (Lenovo IdeaTab A3000-H)
AL lib: Initializing library v1.20.1-253773b3 master AL lib: Supported backends: oboe, null AL lib: Loading config /etc/openal/alsoft.conf... AL lib: Loading config /etc/xdg/alsoft.conf... AL lib: Got binary: /system/bin, app_process AL lib: Loading config /system/bin/alsoft.conf... AL lib: Key disable-cpu-exts not found AL lib: Extensions: +NEON AL lib: Key rt-prio not found AL lib: Key resampler not found AL lib: Key trap-al-error not found AL lib: Key trap-alc-error not found AL lib: Key reverb/boost not found AL lib: Key drivers not found AL lib: Initialized backend "oboe" AL lib: Added "oboe" for playback AL lib: Initialized backend "null" AL lib: No capture backend available! AL lib: Key excludefx not found AL lib: Key default-reverb not found AL lib: Key channels not found AL lib: Key sample-type not found AL lib: Key frequency not found AL lib: Key period_size not found AL lib: Key periods not found AL lib: Key sources not found AL lib: Key slots not found AL lib: Key sends not found AL lib: Key ambi-format not found AL lib: Created device 0x54bba0b0, "" AL lib: ALC_FREQUENCY = 44100 AL lib: Key frequency not found AL lib: Key period_size not found AL lib: Key periods not found AL lib: Key sources not found AL lib: Key sends not found AL lib: Key hrtf not found AL lib: Pre-reset: Stereo, Float, *44100hz, 882 / 2646 buffer AL lib: Got stream with properties: StreamID: 0x54bcf2f0 DeviceId: 0 Direction: Output API type: OpenSLES BufferCapacity: 384 BufferSize: 384 FramesPerBurst: 192 FramesPerCallback: 192 SampleRate: 44100 ChannelCount: 2 Format: I16 SharingMode: Shared PerformanceMode: None CurrentState: Open XRunCount: 0 FramesRead: 0 FramesWritten: 0 AL lib: Post-reset: Stereo, Signed Short, 44100hz, 441 / 882 buffer AL lib: Key stereo-mode not found AL lib: Key cf_level not found AL lib: Key stereo-encoding not found AL lib: Stereo rendering AL lib: Channel config, Main: 3, Real: 2 AL lib: Allocating 5 channels, 20480 bytes AL lib: Enabling first-order ambisonic decoder AL lib: Max sources: 256 (255 + 1), effect slots: 64, sends: 2 AL lib: Front stablizer disabled AL lib: Key dither not found AL lib: Key dither-depth not found AL lib: Dithering enabled (16-bit, 32768) AL lib: Key output-limiter not found AL lib: Output limiter enabled, -0.0005dB limit AL lib: Fixed device latency: 997732ns AL lib: Increasing allocated voices to 256 AL lib: Key volume-adjust not found AL lib: Created context 0x54bce4d0 AL lib: Querying error state on null context (implicitly 0xa004)
And one more log, taken from a really crappy tablet (Lenovo IdeaTab A3000-H)
How did that one perform? What was the test like (how many sources, were they or the listener continuously updated, was reverb used and regularly updated, ...)?
BufferSize: 384 FramesPerBurst: 192
These are quite low; 4.3ms "burst"/period size and 8.7ms buffer size. OpenAL Soft's default period size is 20ms, more than double the total buffer size there. On the one hand, it's nice to hear it manages to stand up to the task, at least to some degree, but on the other it raises some potential issues. For one, it's trying to process updates nearly 230 times a second, which is extreme overkill. Second, when there are updates it doesn't have much time to do them (only 4 milliseconds to process pending updates for sources and effects, and render 192 samples from all sources and effects, or else it underruns). For another, it doesn't have much time to fade between states. When the position or distance changes, it interpolates the panning or gain over the course of a decently sized period (normally 800~1000 samples or so), so on the next mix it potentially has a new position or distance and creates a smooth, continuous transition while things move and play sound.
There are things I might be able to do to deal with that problem, but I'd need to know if it can truly withstand a full update cycle (several dozen sources and at least one reverb effect updated at the same time) and mix in the short time-sensitive callback.
Well, I didn't see any issues with the sound on that tablet. The game doesn't run too many sound sources at the same time though, so I am not sure how good of a test it is. Typically it has 3-4 positional sources playing a static sound in loop, 1 non positional emitter playing music and ~10-20 other positional source that plays every now and then (shots, hits, cheers etc). The sound listener is attached to the game camera which moves constantly, so it updates its position each frame. The is no reverb or any other effects though. Is there a way I can enable some form of "buffer health" reporting so I can give more detailed information about it? If there is, I believe I can easily setup multiple scenes with 10,20,30,40 .. sources that moves constantly or something alone these lines, and give you more more precise data of what's going on. I guess I can attach a reverb or two as well, if I figure out how to do it.
Is there a way I can enable some form of "buffer health" reporting so I can give more detailed information about it?
Not with OpenAL Soft specifically. You can perhaps try a performance analyzer like perf to see how much time the mixing functions take, though I don't know if that works on ARM or if there's an alternative.
I'm not sure how to report the "health" of playback from a periodic asynchronous callback. More specifically, what to report, since the callback can run any number of times in between the app checking. It could be nice to do, but I don't know what would be meaning/useful to give back to the app.
I guess I can attach a reverb or two as well, if I figure out how to do it.
The simplest way would be to set the ALSOFT_DEFAULT_REVERB env var to something like Hangar when you run the app, which will force a reverb preset to be applied. It won't check to see how well updates are handled for it, but it will at least process one when mixing. Alternatively, you can check the alreverb example to see how to set up a reverb effect.
Any plans on having Oboe capture device support? Android 11 start deprecating OpenSLES for native audio so we can't really depend on OpenSLES (for capture device) on future Android versions.
On unrelated note, there seem to be update to Oboe 1.5 which adds few changes and deprecating few things. You may want to check it out.
Is there a documentation regarding what has changed and deprecated? Are there any recording examples for Oboe?
Hmm, this is odd. It seems Google reversed their decision to deprecate OpenSL. This https://github.com/BelledonneCommunications/linphone-android/issues/1150#issuecomment-685708846 references about OpenSL deprecation links to Android 11 features but it seems part of deprecating OpenSL is completely removed from there.
For recording example, you probably can take a look at here: https://github.com/google/oboe/tree/master/samples/LiveEffect
EDIT: Found the OpenSL deprecation notice https://github.com/android/ndk/wiki/Changelog-r21#r21d
Commit 63f5240db253f0ea942a79da3a5f54e7338df493 adds capture support to Oboe. I haven't tested, and Oboe support in general is still considered experimental, but it's there to try out.
Recording is fine, but as soon as I stop the recording, there's al::backend_exception with Failed to pause stream: ErrorUnimplemented message.
That should be fixed with commit b8a4ebd5943e554c7b9f28874f6308d3ff402e15. Though I'm not sure if calling Oboe's stop() method drops any unread samples or not (which would cause incorrect OpenAL capture behavior, if it does drop them).
Hmm, looks like unread samples are dropped, so the samples needs to be read and stored somewhere first just before stopping the device.
Oboe has been supported for some time and seems to be working well. If you find any problems, feel free to open a new issue.