No ability to capture remote audio samples?

[wseemann]

trafficstars

Hello, thank you for this awesome library! I'm trying to implement recording of a remote WebRTC stream. I'm able to capture raw video frames using the addSink method on VideoTrack and write them to a file. There doesn't appear a similar method on AudioTrack. Does this library provide the ability to process remote audio sample? Thanks in advance.

[skydoves]

Hey @wseemann, audio doesn't work by adding a track on the Sink, which renders Video streams.

You can refer to handing audio tracks from Stream's Video SDK.

[wseemann]

Hello @skydoves, thank you for the reply. I looked over the sample you provided but this doesn't seem to address the issue outlined in my original question. The code snippet provided combines multiple media tracks into one. What I'm looking to do is capture the audio samples being sent by the server so I can save them to a file. As I noted, it is possible using addSink on the video track, but there doesn't appear to be a method to retrieve audio track samples. Is there something I'm misunderstanding in this snippet which allows me to do that?

[skydoves]

Hey @wseemann, ah I got your question. Thanks for the details! Do you mean the audio sample callback listeners like this?

• https://github.com/GetStream/stream-video-android/blob/ddea2579360cb09223c7f3f36dda22fd4d5fb3cb/stream-video-android-core/src/main/kotlin/io/getstream/video/android/core/call/connection/StreamPeerConnectionFactory.kt#L207
• https://github.com/GetStream/stream-video-android/blob/ddea2579360cb09223c7f3f36dda22fd4d5fb3cb/stream-video-android-core/src/main/kotlin/io/getstream/video/android/core/call/connection/StreamPeerConnectionFactory.kt#L210

[shenbengit]

@wseemann This library may be what you need，WebRTCExtension use:

val audioDeviceModule = JavaAudioDeviceModule.builder(this)
     .setAudioTrackStateCallback(object : JavaAudioDeviceModule.AudioTrackStateCallback {
        override fun onWebRtcAudioTrackStart() {
            //kotlin
            audioDeviceModule.setAudioTrackSamplesReadyCallback {
                val audioFormat = it.audioFormat
                val channelCount = it.channelCount
                val sampleRate = it.sampleRate
                //pcm
                val data = it.data
            }
        }

        override fun onWebRtcAudioTrackStop() {
        }

}) 
.setSamplesReadyCallback {

}.createAudioDeviceModule()

[zibozeng]

@skydoves I'm also looking for the solution for detecting whether the remote user is talking. For what you posted above, I checked. I saw the comments for these two callbacks, says they are for detecting audio input, does it mean the local audio input or remote? since what i want to know is how to detect the remote users. What's more is this callback doesn't include any information about which audio track the data was coming through, so how do I detect which remote user is talking if there are multiple connections communicating around?

[tessmerandre]

@zibozeng have you found any solution for this? If so, do you mind sharing, please?