Direct output support for non-MMAP path

[nift4]

Hi, apologies if this is the wrong issue tracker.

Qualcomm and some OEMs (not supported on Tensor Pixel devices, though) usually provide a special output mix port not having the offload flag (i.e. not the PCM offload support recently added), and not having the MMAP flag either, but a pure direct (PCM-only) output. This output is interesting due to inherently different behaviour of the port, for example no effects, more sample rate / format choices (while MMAP usually has very limited or only one choice on the same device) and sometimes even reconfiguring the output sound card. However, there's currently no public API to access this. Using dlopen() aside of blessed stable APIs is a major pain, and I'd like to get rid of that in the future.

Hence, it'd be awesome if AAudio supported direct output tracks for non-MMAP (MMAP is always direct as far as HAL is concerned, I believe), and an API to query if the created track is indeed direct (audio policy will silently fall back to non-direct, but AudioTrack::flags() & AUDIO_OUTPUT_FLAG_DIRECT is a sufficient check). There's already a public API to query for this since Android Q, only playback is missing. I understand this cannot be properly backported in Oboe, but at least future Android version supports in AAudio and Oboe would be awesome to have.

I think it only requires modifying this: https://cs.android.com/android/platform/superproject/main/+/main:frameworks/av/media/libaaudio/src/legacy/AudioStreamTrack.cpp;l=81;drc=353f31abbc5c72706d1498bb3763f7be2c11abbc

to pass AUDIO_OUTPUT_FLAG_DIRECT (without offload), similar to the offload support that was recently added.

Thanks for considering this request!

[flamme]

AFAIK, the direct PCM on qcom and other OEMs bypass the data processing(data conversion, volume control, etc) on the framework side. But it doesn't bypass the HAL or AoC's data processing. That means the data can still be converted at AoC. For instance, the direct PCM mix port export capabilities for 44.1kHz PCM and can routed to USB device while the USB device only supports 48kHz, the sample rate conversion will still happen on AoC.

What is more, currently, the direct PCM works with some customized modification from the OEMs side and also bring some uncertain behaviors when having multiple clients active simultaneously.

From the framework side, the data conversion will only happen if the client's playback configuration is different from the mix port's capabilities. The effect is only attached if the client creates it or it is a global effect. That is more like the question #2193 about the device capabilities.

I'd like to ask what is the scenarios for the direct PCM. Is it for media playback on peripheral or built-in speaker? What is the common configuration that will trigger the data conversion happen from the audio framework?

[flamme]

BTW, the framework is automatically use direct if it is not PCM. https://cs.android.com/android/platform/superproject/+/android15-qpr2-release:frameworks/av/media/libaudioclient/AudioTrack.cpp;l=482?q=AudioTrack.cpp

[nift4]

I'm concerned about media playback via wired headset, USB and HDMI.

Based on my understanding, qcom has two types of direct playback in their audio HAL:

PCM playback intended for multi-channel on HDMI or "Proxy": https://github.com/TheXPerienceProject/android_hardware_qcom_audio/blob/9e8f9772fa497a76d0fbaa664d2ef39a13328b4b/hal/audio_hw.c#L2926

(but audio policy declares that this supports stereo as well)

and "Direct PCM" aka PCM offload to do resampling, mixing, etc on ADSP for power saving: https://github.com/TheXPerienceProject/android_hardware_qcom_audio/blob/9e8f9772fa497a76d0fbaa664d2ef39a13328b4b/hal/audio_hw.c#L2978

I'm not knowledgable enough about different QCOM audio paths to tell when it will just resample on ADSP and when we actually get higher quality, but on some devices it's a selling point that internal DAC supports Hi-Res (LG Quad DAC anyone?), and using direct playback is required to get something above 16-bit 48000 into the audio HAL on atleast one such device I got to test. Even if ADSP resamples on most devices, it's a net win anyway though, because it saves power, and for those few devices where I know it's needed, I'm doing exactly that.

[flamme]

Thanks for the explanation!

For the USB case, it can be solved by [AudioManager.setPerferredMixerAttributes], which allows to config the mixer in the framework. If it is mixer behavior is bit-perfect, it can also avoid data conversion from the HAL and AoC side.

I'm not knowledgable enough about different QCOM audio paths to tell when it will just resample on ADSP and when we actually get higher quality

For qcom, I'm not familiar with their implementation either and it is also confidential. Also note from the link shared above, the flag for direct PCM is AUDIO_OUTPUT_FLAG_DIRECT_PCM, which is different from AUDIO_OUTPUT_FLAG_DIRECT .

but on some devices it's a selling point that internal DAC supports Hi-Res (LG Quad DAC anyone?), and using direct playback is required to get something above 16-bit 48000 into the audio HAL on atleast one such device I got to test.

That sounds like the framework is not choosing the right mix port and unnecessary data conversion is involved or the device is not exposing its capabilities correctly to me.

Even if ADSP resamples on most devices, it's a net win anyway though, because it saves power, and for those few devices where I know it's needed, I'm doing exactly that.

I love saving power. I dislike unnecessary conversion no matter from power saving or audio quality point of view.

Bring availability to explicitly request direct playback will need some more discussion about the props and cons. I think it is also worth mentioned that PCM offload is supported by the framework since Android 16, which is also a kind of direct. It is expected to see more device support it in the future.

[nift4]

Hi @flamme , thanks for the reply!

Regarding setPreferredMixerAttributes, I already have experimented with it before and I am very happy about this feature, I'm not yet sure if I can ship it in production across all devices at this moment though because as usual, OEMs introduce bugs and I found a quite servere one. On my MTK test device on Android 14, MTK Hi-Fi presumably conflicts with it and I was able to get audioserver in a busy loop with using 100% cpu which eventually crashed the whole system. I'll make sure to open a seperate bug report if I manage to reproduce it properly, but I suspect there's no way this can be fixed on devices already shipped (my specific test device will probably never get OTA to A15 for example) :/ additionally on Pixel 7a Android 14 QPR3 setting 88200Hz sample rate causes chipmunk audio because audio HAL does not change usb device sample rate from 96000Hz, but I did not yet upgrade it to Android 15 or 16 due to unrelated reasons so I'm holding off reporting that until I can upgrade (and test if it may already be fixed). That aside, the feature is great so I hope it will be more stable in the ecosystem in the future.

Regarding what QCOM did, luckily QCOM releases some parts of Audio HAL and Audio Policy (especially older chipset versions) freely under Apache2.0 license at https://git.codelinaro.org/clo/la (you can see more - although slightly outdated - information at: https://wiki.codelinaro.org/en/clo/la/overview) I noticed I was looking at outdated audio HAL after you mentioned direct PCM flag, because they have changed it to just DIRECT back in 2017: https://github.com/CalyxOS/platform_hardware_qcom-caf_sm8350_audio/commit/ac3415870d63b18694e49d982e89ccb10dbcb2c3 so there is quite a lot of currently shipped devices that have direct flag used for "PCM offload" usecase before AOSP supported PCM offload officially. If Android 16 supports PCM offload as public API (especially through AAudio) and QCOM switches to this method, it would be amazing. (I'm not personally aware of MTK or SLSI or any of the big OEMs using direct mixPort aside of using QCOM solution, so that would cover most of the need for this feature I believe.)

Of course it would still be nice to have direct output because I guess it'll be impossible to get every vendor to switch to PCM offload (and it may not actually apply to every usecase either - point in case: Amlogic TV boxes).

That sounds like the framework is not choosing the right mix port and unnecessary data conversion is involved or the device is not exposing its capabilities correctly to me.

The device where I can personally confirm this is a quite old LG V20 running Android 8.0 - the primary mix is fixed to 48000 at AUDIO_FORMAT_PCM_16_BIT: https://dumps.tadiphone.dev/dumps/lge/elsa/-/blob/elsa_nao_us-user-8.0.0-OPR1.170623.032-183241436cd40-release-keys/system/vendor/etc/audio_policy.conf?ref_type=heads#L25 While the direct port supports a lot more formats: https://dumps.tadiphone.dev/dumps/lge/elsa/-/blob/elsa_nao_us-user-8.0.0-OPR1.170623.032-183241436cd40-release-keys/system/vendor/etc/audio_policy.conf?ref_type=heads#L53 The behaviour in this conf can be experimentally confirmed in running system. It makes sense that this occurs because audio policy will not choose direct port if it's not requested and not required for successful playback, and setting primary output to 48000 int16 probably was a compatibility choice as a large majority of devices has it configured like that. The device according to spec sheet (https://www.lg.com/ca_en/cell-phones/v20/), built-in music player app and some paid third party players however supports 24 bit 384kHz. The only way to output that is using the DIRECT port.

I can see similar definitions in way more modern devices as well (for example QCOM Xiaomi Android 13): https://dumps.tadiphone.dev/dumps/xiaomi/marble/-/blob/missi_phone_cn-user-13-TKQ1.221114.001-V14.0.12.0.TMRCNXM-release-keys/vendor/etc/audio/sku_parrot/audio_policy_configuration.xml?ref_type=heads#L121 But I do not have physical access to confirm it on this device, so I do not want to base my argumentation on that to avoid wasting your time ;)

Bring availability to explicitly request direct playback will need some more discussion about the props and cons.

I assume you are talking about Google-internal discussion, in which I just want to add that some paid apps currently use dlopen() of libaudioclient by bypassing linker namespace and accessing AudioTrack.cpp to pass in direct flag, for example com.maxmpz.audioplayer (before A15) or com.extreamsd.usbaudioplayerpro or com.neutroncode.mp, if you enable it in the app settings. So it's less a matter of "make it possible" because it already is, but just about if it will be blessed through public API. Less private API reliance is good for everyone :)

[flamme]

Regarding setPreferredMixerAttributes, I already have experimented with it before and I am very happy about this feature, I'm not yet sure if I can ship it in production across all devices at this moment though because as usual, OEMs introduce bugs and I found a quite servere one. On my MTK test device on Android 14, MTK Hi-Fi presumably conflicts with it and I was able to get audioserver in a busy loop with using 100% cpu which eventually crashed the whole system. I'll make sure to open a seperate bug report if I manage to reproduce it properly, but I suspect there's no way this can be fixed on devices already shipped (my specific test device will probably never get OTA to A15 for example) :/ additionally on Pixel 7a Android 14 QPR3 setting 88200Hz sample rate causes chipmunk audio because audio HAL does not change usb device sample rate from 96000Hz, but I did not yet upgrade it to Android 15 or 16 due to unrelated reasons so I'm holding off reporting that until I can upgrade (and test if it may already be fixed). That aside, the feature is great so I hope it will be more stable in the ecosystem in the future.

The unfortunate part of the feature is that the mixer behavior can be DEFAULT or BIT_PERFECT. BIT_PERFECT will require change from the HAL and AoC side. DEFAULT will not. For devices support USB playback, they will support preferred mixer attributes with DEFAULT mixer behavior when upgrading to API 34 or later. But note that with mixer behavior DEFAULT, it is only guaranteed the framework tell the HAL to open the hifi mix port at the requested configuration, but the HAL may not open the USB device at the same configuration, they can do their own conversion. Ideally, it would be great if all OEMs can support bit-perfect. For the issue you are seeing, please feel free to file bugs on MTK and Pixel.

If Android 16 supports PCM offload as public API (especially through AAudio) and QCOM switches to this method, it would be amazing. (I'm not personally aware of MTK or SLSI or any of the big OEMs using direct mixPort aside of using QCOM solution, so that would cover most of the need for this feature I believe.)

I cannot put my word for QCOM's plan. Offload PCM is official support on Android 16, all SoC vendors adoption will be relied on their plans. I totally understand the benefit of direct PCM, but there are also some drawback so that it needs some more discussion about making it official support.

I assume you are talking about Google-internal discussion, in which I just want to add that some paid apps currently use dlopen() of libaudioclient by bypassing linker namespace and accessing AudioTrack.cpp to pass in direct flag, for example com.maxmpz.audioplayer (before A15) or com.extreamsd.usbaudioplayerpro or com.neutroncode.mp, if you enable it in the app settings. So it's less a matter of "make it possible" because it already is, but just about if it will be blessed through public API. Less private API reliance is good for everyone :)

Thanks for sharing the extra information! That will be helpful for the internal discussion.