Audio sampling and choosing an audio output device

[tak-hntlabs]

I really like how easy it is to set up and get my app working with video & audio calls. I have two API questions.

1. How do I get the audio stream for each participant? I would like to sample the audio stream and do further audio processing.

2. How can I set a specific audio output device? I would like to show the user a list of audio output devices (BT devices, Speaker, and Phone earpiece). When the user picks one, then route sound to that audio output. Currently. when Bluetooth is connected, the audio is always routed to the default BT device. When BT is off, then setting toggleSoundSetup( true | false ) switches between Speaker and the Phone's earpiece. It doesn't work when BT is connected.

Environment

• Node.js version: 15.6.2
• React Native version: 0.64.2
• React Native platform + platform version: iOS 14

react-native-twilio-video-webrtc

Version: yarn 1.22.10

[slycoder]

1.) How do you plan on doing the audioprocessing? If you're willing to dive into the iOS code, you can call addSink on the track you want to listen in on and implement your own custom sink: https://twilio.github.io/twilio-video-ios/docs/latest_4.x/Protocols/TVIAudioSink.html#//api/name/renderSample:.

2.) Ah yes, there is funky interaction between difference devices. It's not very clean right now and one of these days I'd like to rewrite it to provide that control. If you wanna take a crack at that, it would be awesome

[tak-hntlabs]

RE: audio processing, I'm not sure how to do it yet. As a first step, would like to have access to the audio stream. I'll take a look at addSink. I'll keep in mind to make the solution generic so that there is a way to make it back to the library.

RE: interaction between different audio output devices. Sure, I can take a crack at it. I've built an audio experience with existing behavior (described above), and it's good enough for us at the moment. Will work on more granular audio output control later, after I get everything else on my priority list working :)

[yadavmurari111]

@slycoder any update on the part where i am not able toggle audio output between speaker and earpiece .

I am using audio call feature using

const onConnectTwilio = token => { twilioRef.current.connect({ accessToken: token, enableVideo: false, enableAudio: true, };

this way i am entering the room , everything works fine apart from 2 issues.

1. toggleSoundSetup not working for me to toggle audio output.
2. After joining room , on one device speaker is having audio output while on other device earpiece is having audio output.

const toggleSoundSetup = speakerMode => { twilioRef.current.toggleSoundSetup(speakerMode); };

@slycoder please help me out.

[Suresha-Manjunatha]

2. After joining room , on one device speaker is having audio output while on other device earpiece is having audio output.

@slycoder please help me out.

I too having this issue please help to resolve this issue

[yadavmurari111]

@Suresha-Manjunatha you can call toggleSoundSetup(isSpeakerMode) on button press code for reference

const [isSpeakerMode, setIsSpeakerMode] = useState(true);

const toggleSoundSetup = speakerMode => { twilioRef?.current?.toggleSoundSetup(speakerMode); setIsSpeakerMode(!speakerMode); };

[Suresha-Manjunatha]

@yadavmurari111 this solution not working, twilioRef not having "current" object or not even toggelSoundSetup() function

twiolioRef contains only just this

{ "_reactInternalInstance": {

}, "_reactInternals": { "_debugHookTypes": null, "_debugID": 24711, "_debugNeedsRemount": false, "_debugOwner": { "_debugHookTypes": null, "_debugID": 24673, "_debugNeedsRemount": false, "_debugOwner": [ FiberNode ], "_debugSource": [ Object ], "actualDuration": 21.859843730926514, "actualStartTime": 1258230381.069722, "alternate": [ FiberNode ], "child": [ FiberNode ], "childLanes": 0, "dependencies": null, "elementType": [ FunctionVideoCall ], "firstEffect": [ FiberNode ], "flags": 5, "index": 0, "key": null, "lanes": 1, "lastEffect": [ FiberNode ], "memoizedProps": [ Object ], "memoizedState": [ Object ], "mode": 8, "nextEffect": null, "pendingProps": [ Object ], "ref": null, "return": [ FiberNode ], "selfBaseDuration": 18.559268951416016, "sibling": null, "stateNode": [ VideoCall ], "tag": 1, "treeBaseDuration": 21.719770193099976, "type": [ FunctionVideoCall ], "updateQueue": [ Object ] }, "_debugSource": { "columnNumber": 7, "fileName": "/data/home/ubuntu/Desktop/ylanes1-187075-react-native/src/packages/blocks/VideoCallModule/src/Screen/VideoCallController.tsx", "lineNumber": 1850 }, "actualDuration": 0.29265475273132324, "actualStartTime": 1258230400.715337, "alternate": { "_debugHookTypes": null, "_debugID": 24711, "_debugNeedsRemount": false, "_debugOwner": [ FiberNode ], "_debugSource": [ Object ], "actualDuration": 0.2679622173309326, "actualStartTime": 1258230428.153952, "alternate": [ Circular ], "child": [ FiberNode ], "childLanes": 0, "dependencies": null, "elementType": [ FunctionCustomTwilioVideoView ], "firstEffect": [ FiberNode ], "flags": 1, "index": 1, "key": null, "lanes": 0, "lastEffect": [ FiberNode ], "memoizedProps": [ Object ], "memoizedState": null, "mode": 8, "nextEffect": null, "pendingProps": [ Object ], "ref": [ Functionref ], "return": [ FiberNode ], "selfBaseDuration": 0.23100018501281738, "sibling": null, "stateNode": [ Circular ], "tag": 1, "treeBaseDuration": 0.25300002098083496, "type": [ FunctionCustomTwilioVideoView ], "updateQueue": [ Object ] }, "child": { "_debugHookTypes": null, "_debugID": 24713, "_debugNeedsRemount": false, "_debugOwner": [ Circular ], "_debugSource": [ Object ], "actualDuration": 0.034538984298706055, "actualStartTime": 1258230400.972798, "alternate": [ FiberNode ], "child": null, "childLanes": 0, "dependencies": null, "elementType": "RNCustomTwilioVideoView", "firstEffect": null, "flags": 4, "index": 0, "key": null, "lanes": 0, "lastEffect": null, "memoizedProps": [ Object ], "memoizedState": null, "mode": 8, "nextEffect": null, "pendingProps": [ Object ], "ref": [ Functionref ], "return": [ Circular ], "selfBaseDuration": 0.024576902389526367, "sibling": null, "stateNode": [ ReactNativeFiberHostComponent ], "tag": 5, "treeBaseDuration": 0.024576902389526367, "type": "RNCustomTwilioVideoView", "updateQueue": null }, "childLanes": 0, "dependencies": null, "elementType": [ FunctionCustomTwilioVideoView ], "firstEffect": { "_debugHookTypes": null, "_debugID": 24713, "_debugNeedsRemount": false, "_debugOwner": [ Circular ], "_debugSource": [ Object ], "actualDuration": 0.034538984298706055, "actualStartTime": 1258230400.972798, "alternate": [ FiberNode ], "child": null, "childLanes": 0, "dependencies": null, "elementType": "RNCustomTwilioVideoView", "firstEffect": null, "flags": 4, "index": 0, "key": null, "lanes": 0, "lastEffect": null, "memoizedProps": [ Object ], "memoizedState": null, "mode": 8, "nextEffect": null, "pendingProps": [ Object ], "ref": [ Functionref ], "return": [ Circular ], "selfBaseDuration": 0.024576902389526367, "sibling": null, "stateNode": [ ReactNativeFiberHostComponent ], "tag": 5, "treeBaseDuration": 0.024576902389526367, "type": "RNCustomTwilioVideoView", "updateQueue": null }, "flags": 1, "index": 1, "key": null, "lanes": 0, "lastEffect": { "_debugHookTypes": null, "_debugID": 24713, "_debugNeedsRemount": false, "_debugOwner": [ Circular ], "_debugSource": [ Object ], "actualDuration": 0.034538984298706055, "actualStartTime": 1258230400.972798, "alternate": [ FiberNode ], "child": null, "childLanes": 0, "dependencies": null, "elementType": "RNCustomTwilioVideoView", "firstEffect": null, "flags": 4, "index": 0, "key": null, "lanes": 0, "lastEffect": null, "memoizedProps": [ Object ], "memoizedState": null, "mode": 8, "nextEffect": null, "pendingProps": [ Object ], "ref": [ Functionref ], "return": [ Circular ], "selfBaseDuration": 0.024576902389526367, "sibling": null, "stateNode": [ ReactNativeFiberHostComponent ], "tag": 5, "treeBaseDuration": 0.024576902389526367, "type": "RNCustomTwilioVideoView", "updateQueue": null }, "memoizedProps": { "onDominantSpeakerDidChange": [ Functionanonymous ], "onNetworkQualityLevelsChanged": [ Functionanonymous ], "onParticipantAddedVideoTrack": [ Functionanonymous ], "onParticipantDisabledAudioTrack": [ Functionanonymous ], "onParticipantDisabledVideoTrack": [ Functionanonymous ], "onParticipantEnabledAudioTrack": [ Functionanonymous ], "onParticipantEnabledVideoTrack": [ Functionanonymous ], "onParticipantRemovedVideoTrack": [ Functionanonymous ], "onRoomDidConnect": [ Functionanonymous ], "onRoomDidDisconnect": [ Function_callee26 ], "onRoomDidFailToConnect": [ Functionanonymous ], "onRoomParticipantDidConnect": [ Functionanonymous ], "onRoomParticipantDidDisconnect": [ Function_callee27 ], "onStatsReceived": [ Functionanonymous ] }, "memoizedState": null, "mode": 8, "nextEffect": null, "pendingProps": { "onDominantSpeakerDidChange": [ Functionanonymous ], "onNetworkQualityLevelsChanged": [ Functionanonymous ], "onParticipantAddedVideoTrack": [ Functionanonymous ], "onParticipantDisabledAudioTrack": [ Functionanonymous ], "onParticipantDisabledVideoTrack": [ Functionanonymous ], "onParticipantEnabledAudioTrack": [ Functionanonymous ], "onParticipantEnabledVideoTrack": [ Functionanonymous ], "onParticipantRemovedVideoTrack": [ Functionanonymous ], "onRoomDidConnect": [ Functionanonymous ], "onRoomDidDisconnect": [ Function_callee26 ], "onRoomDidFailToConnect": [ Functionanonymous ], "onRoomParticipantDidConnect": [ Functionanonymous ], "onRoomParticipantDidDisconnect": [ Function_callee27 ], "onStatsReceived": [ Functionanonymous ] }, "ref": [ Functionref ], "return": { "_debugHookTypes": null, "_debugID": 24693, "_debugNeedsRemount": false, "_debugOwner": [ FiberNode ], "_debugSource": null, "actualDuration": 0.3421156406402588, "actualStartTime": 1258230400.664414, "alternate": [ FiberNode ], "child": [ Circular ], "childLanes": 0, "dependencies": null, "elementType": null, "firstEffect": [ FiberNode ], "flags": 0, "index": 0, "key": null, "lanes": 0, "lastEffect": [ FiberNode ], "memoizedProps": [ Array ], "memoizedState": null, "mode": 8, "nextEffect": null, "pendingProps": [ Array ], "ref": null, "return": [ FiberNode ], "selfBaseDuration": 0.047499895095825195, "sibling": [ FiberNode ], "stateNode": null, "tag": 7, "treeBaseDuration": 0.32538461685180664, "type": null, "updateQueue": null }, "selfBaseDuration": 0.2533078193664551, "sibling": null, "stateNode": [ Circular ], "tag": 1, "treeBaseDuration": 0.27788472175598145, "type": [ FunctionCustomTwilioVideoView ], "updateQueue": { "baseState": null, "effects": null, "firstBaseUpdate": null, "lastBaseUpdate": null, "shared": [ Object ] } }, "context": {

}, "props": { "onDominantSpeakerDidChange": [ Functionanonymous ], "onNetworkQualityLevelsChanged": [ Functionanonymous ], "onParticipantAddedVideoTrack": [ Functionanonymous ], "onParticipantDisabledAudioTrack": [ Functionanonymous ], "onParticipantDisabledVideoTrack": [ Functionanonymous ], "onParticipantEnabledAudioTrack": [ Functionanonymous ], "onParticipantEnabledVideoTrack": [ Functionanonymous ], "onParticipantRemovedVideoTrack": [ Functionanonymous ], "onRoomDidConnect": [ Functionanonymous ], "onRoomDidDisconnect": [ Function_callee26 ], "onRoomDidFailToConnect": [ Functionanonymous ], "onRoomParticipantDidConnect": [ Functionanonymous ], "onRoomParticipantDidDisconnect": [ Function_callee27 ], "onStatsReceived": [ Functionanonymous ] }, "refs": { "videoView": { "_children": [ Array ], "_internalFiberInstanceHandleDEV": [ FiberNode ], "_nativeTag": 1147, "viewConfig": [ Object ] } }, "state": null, "updater": { "enqueueForceUpdate": [ FunctionenqueueForceUpdate ], "enqueueReplaceState": [ FunctionenqueueReplaceState ], "enqueueSetState": [ FunctionenqueueSetState ], "isMounted": [ FunctionisMounted ] } }

[Suresha-Manjunatha]

@yadavmurari111 could you please look into this issue, since the twilioRef was not having current object and toggleSoundSetup in my case, the solution you shared is not useful, so could you please tell, why the twilioRef is not having these things, or any other solution to solve this audio issue.