Multi-channel audio not received by AudioOutput component on Windows

[OlafZwe]

Describe the bug On Windows 10, using Media Foundation with Audio Mode Unity and deselecting Stereo in Unity Audio settings. When playing a video with a multi-channel audio track (e.g. 4 channels), the AudioOutput component attached to the media player only receives silence in its OnAudioFilterRead function. The audio is still audible, but appears to be playing in System Direct mode rather than through Unity. See screenshot below for setup.

We found this bug while developing a custom AudioOutput component that should take the raw multi-channel audio data and pass it through to the Audio360 SDK's NativeSpatDecoderQueue in order to be spatialized. To be able to handle (adaptive streaming) spatial audio, we need to be able to access the raw multi-channel audio data directly, regardless of Windows' speaker setup or Unity's speaker/audio setup. Is this possible?

This bug might be related to issue #72 , but that issue did appear to describe a different situation.

Your Setup (please complete the following information):

• Unity version: 2018.2.21f1 (also tested on 2018.1.6f1)
• AVPro Video version: 1.10.0
• Operating system version: Windows 10 Pro (64-bit) May 2019 update (v.1903)
• Device model: N/A
• Video specs (resolution, frame-rate, codec, file size): mp4 video container with H.264 codec and a single 4-channel AAC audio track

To Reproduce

1. Use the GameObject > AVPro Video > Media Player with Unity Audio menu option
2. Change the Media Player (and AudioOutput) settings according to the screenshot below
3. Play a (local) video with multi-channel audio (e.g. 4 channels)
4. Notice that changing the AudioSource settings or disabling the component altogether does not change or stop the audio from playing.

Screenshots

Videos Example video with 4-channel spatial audio

[owlboy]

Hello! I have been struggling with this issue since 2018. As that post states, this appears to be something that was supported in the past. Is this incorrect?

Is there any information that can be shared about this topic?

Can you elaborate on what you said in this comment on #72?

This component might not help in all cases, eg being able to direct audio channels to specific Transforms, but this could act as a reference to make a new component to achieve this.

My Goal:

Sample Videos:

• 5_1-test-ac3-mkv.mkv
• 5_1-test-ac3-mp4.mp4

My Setup:

• Unity version: 2018.4.18f1
• AVPro Video version: 1.10.1t
• Operating system version: Windows 10
• Device model: N/A
• Windows Speaker Config: Same behavior in Stereo and 5.1 modes.
• Video specs (resolution, frame-rate, codec, file size): mp4 video container with H.264 codec and a single 6-channel AC3 audio track
• Video specs (resolution, frame-rate, codec, file size): mkv video container with H.264 codec and a single 6-channel AC3 audio track

thank you!

[AndrewRH]

Thanks both of you for sharing such detailed information. We'll look into this with your files and use cases in mind and report back soon.

[fingerx]

@AndrewRH any news?thanks

[AndrewRH]

@OlafZwe Hi, I know this is rather late, but thanks for the excellent example. We have replicated the issue and are working on a fix.

@owlboy I believe this is a slightly different issue to the original poster, but we're looking into it.

[fingerx]

Hope it can be fixed soon. thank.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. If you wish to continue this issue, then please create a new issue as we do not monitor closed issues.

[fingerx]

@AndrewRH any news for that?

[cross6180]

@AndrewRH I am trying to do this with the Android platform. I get all the 5.1 channel raw data from the GrabAudio() function in the AudioOutputManager.cs, but find out that the data array is arranged differently from Windows (look like it is orderless). Is there something I missed? Thank you.

My Setup: Unity version: 2019.4.10f1 AVPro Video version: v2.1.6 trial version Operating system version: Android 10 Device model: VR HMD Video specs (resolution, frame-rate, codec, file size): mp4 video container with H.264 codec and a single 6-channel audio track

[AndrewRH]

Going to look into this soon for Windows, currently I think it works okay if you're using multi-channel PCM audio. For AAC compressed audio there are internal limits in the Windows decoder for number of channels. I will be investigating further soon.

[Ste-RH]

@AndrewRH I am trying to do this with the Android platform. I get all the 5.1 channel raw data from the GrabAudio() function in the AudioOutputManager.cs, but find out that the data array is arranged differently from Windows (look like it is orderless). Is there something I missed? Thank you.

My Setup: Unity version: 2019.4.10f1 AVPro Video version: v2.1.6 trial version Operating system version: Android 10 Device model: VR HMD Video specs (resolution, frame-rate, codec, file size): mp4 video container with H.264 codec and a single 6-channel audio track

On Android…I believe this was a bug that we fixed a month or so ago. Do grab the latest version and see if the issue is addressed.

[gabrielstuff]

Hello, Any news on this issue ? It would be great to know the status ?

Thanks a lot !

[andriworld]

For anyone trying to play each audio channel of a 5.1 audio track in a separate Unity AudioSource (as in https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/158#issuecomment-594321016) here's how I configured each speaker object:

Tested on Windows and Android, Unity 2021.3.9f1, AVPro v2.6.7.
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 92 kb/s (default)