Hardware.setSpeakerphoneOn() not working on iOS

[janoskranczler]

Hardware.setSpeakerphoneOn() doesn't switch between speaker/earpiece. I tried it on an iPhone and it doesn't work. It works well on Android (also tried).

Flutter version: [✓] Flutter (Channel stable, 3.3.2, on macOS 12.6 21G115 darwin-arm, locale en-HU) • Flutter version 3.3.2 on channel stable at /Users/janoskranczler/flutter • Upstream repository https://github.com/flutter/flutter.git • Framework revision e3c29ec00c (2 weeks ago), 2022-09-14 08:46:55 -0500 • Engine revision a4ff2c53d8 • Dart version 2.18.1 • DevTools version 2.15.0

Plugin version: 1.1.4

OS: iOS

OS version: 16.0.2

[cloudwebrtc]

okay, let me check.

[janoskranczler]

Thank you! Actually, I debugged and I think I found an issue and also a solution: https://github.com/flutter-webrtc/flutter-webrtc/issues/1098

[cloudwebrtc]

Audio management for iOS on flutter-webrtc looks a bit confusing. I am trying to simplify and solve these problems.

[cloudwebrtc]

fixed https://github.com/livekit/client-sdk-flutter/releases/tag/v1.1.6

[cloudwebrtc]

hey @janoskranczler, sorry, I got confused and forgot to update the dependency for flutter-webrtc, I should update it tomorrow (there are some other PRs that need to be tested and verified together) and release a new version。

[janoskranczler]

Got it! Thank you!

[cloudwebrtc]

fixed https://github.com/livekit/client-sdk-flutter/pull/190, will publish a new version to pub today

[janoskranczler]

@cloudwebrtc Great news! Thank you!

[janoskranczler]

@cloudwebrtc I've tested it and it doesn't work. I'm not sure, but I think the problem is that there was no new pod released for this change (1.1.2): https://github.com/livekit/client-sdk-flutter/blob/main/ios/livekit_client.podspec#L3

[cloudwebrtc]

The change happens in flutter-webrtc, you can try running flutter clean && flutter pub get and rm ios/Podfile.lock && pod install

[janoskranczler]

@cloudwebrtc I've tested it with your recommended changes, but it still has not worked on my project. It worked with the example app and I debugged what is the difference and found that when I started a meeting with muted microphone then the speaker/earpiece change will not work. I created a branch for a modified LiveKit example app, which could help to reproduce the error: https://github.com/janoskranczler/client-sdk-flutter/commit/b54088272928e33e05252928ff0b6fa229f645f7

[janoskranczler]

Also, I forgot to mention you can only reproduce the error on a physical device. I don't know why (probably because there is no speaker) but it can't be reproduced on the simulator.

[janoskranczler]

You can reproduce the error if you start the modified example app and start changing the speaker/earpiece with the new button. You should see the following error messages every time when you change to speaker:

[as_client]     AVAudioSession_iOS.mm:2194  Error: category option 'defaultToSpeaker' is only applicable with category 'playAndRecord'
[as_client]     AVAudioSession_iOS.mm:2370  Failed to set category, error: -50
Port override failed due to: Error Domain=NSOSStatusErrorDomain Code=-50 "(null)"

[cloudwebrtc]

Yes, I reproduced the issue you described, but the reason is When we turn off the microphone, we will switch to AVAudioSessionCategoryPlayback mode, this mode does not support Earpiece/Loudspeaker switching, the sound will be fixed in the Loudspeaker

If we still need to switch the Earpiece/Loudspeaker in playback-only mode, we must switch the mode to AVAudioSessionCategoryPlayAndRecord, but the side effect is that the privacy prompt will inform the user that we are using a microphone, even if we do not actually collect data from the microphone, this time to the user cause anxiety.

[janoskranczler]

What do you suggest, how could I fix this issue? I couldn't find a suitable workaround for this yet.

[cloudwebrtc]

First we should get the current Audio Deivce state enum AudioTrackState { none, remoteOnly, localOnly, localAndRemote, }

https://github.com/livekit/client-sdk-flutter/blob/main/lib/src/track/audio_management.dart#L111

When in localOnly, localAndRemote (recordAndPlay), it is possible to switch the earpiece/speaker, while in remoteOnly (playback), we cannot set setSpeakerOn

When in remoteOnly(playback) mode, there are two processing methods

1. Ignore the setSpeakerOn operation and let iOS handle it automatically. At this time, the mic privacy indicator will not light up.

2. Modify RTCAudioSession and change the category to AVAudioSessionCategoryPlayAndRecord, then setSpeakerOn/Off will become effective, but the mic privacy indicator will be on.

It depends on whether the user holds the handset to his ear when he is not talking.

I think in video conference mode we should use option 1, And in audio mode, it seems option 2 way is better

[janoskranczler]

What do you mean by option 1? Could the user still control the speaker/earpiece switch from the interface?

[cloudwebrtc]

in option1, The speaker/earpiece should not be able to switch manually, but it should be possible to switch between wired headphones/Bluetooth and the speaker

[janoskranczler]

@cloudwebrtc I think it could be solved if you could set the port override here: https://github.com/livekit/client-sdk-flutter/blob/84b9ef130fedafa1eeed9cf82689649183a69b85/lib/src/track/audio_management.dart#L21

Currently, the NativeAudioConfiguration could set category, categoryOptions and mode, but not port override: https://github.com/livekit/client-sdk-flutter/blob/56aeacc291af3e6ab8a32128263798e5734abc03/lib/src/support/native_audio.dart#L71

If I could set the port override here LiveKit could control the audio category between playback and playAndRecord category and also it could be overridden with port override by a user.

[janoskranczler]

Meanwhile, I found that you can set the portoverride or the defaultToSpeaker option only on playAndRecord category and this won't help with the microphone issue either.

I still do not understand why the microphone will still be used when I disable the microphone.

[davidzhao]

@janoskranczler if you are using playAndRecord, the underlying stack will have to acquire record permissions. On iOS, the WebRTC stack activates the recording stack in that mode.

This is the dilemma:

• if you are in playAndRecord, then it will allow you to playback to defaultToSpeaker, but it will keep microphone indicator on
• if you are in playback, it will not allow you to defaultToSpeaker. apple docs

[janoskranczler]

@cloudwebrtc @davidzhao Thanks for the explanation. I've collected 3 issues according to the mic and earpiece/loudspeaker problem:

1. There should be an option for the user to completely disable the track-based audio configuration in LiveKit: https://github.com/livekit/client-sdk-flutter/blob/84b9ef130fedafa1eeed9cf82689649183a69b85/lib/src/track/audio_management.dart#L84-L83
2. ~I couldn't figure out why, but~ the loudspeaker switch only works after the microphone is initialized (at least once: setMicrophoneEnabled(true)). After the initialization, even if you turn it off the switch still works. (you can check on my fork's branch: https://github.com/janoskranczler/client-sdk-flutter/commit/b54088272928e33e05252928ff0b6fa229f645f7) Update: I found out why the loudspeaker only works when the microphone was initialized. It is because of the same track-based audio configuration but in flutter-webrtc and it's only initialized with playAndRecord category when there is at least one local audio track.
3. I've created a new issue for this one (https://github.com/livekit/client-sdk-flutter/issues/196), but I've tested with the LiveKit example app (current main branch without any modification: https://github.com/livekit/client-sdk-flutter/commit/22a18d3e9cff352c56870aa723567777855bbf2b) and found that the microphone indicator won't turn off after I mute the microphone. (The camera indicator works as expected)

Update: According to the playAndRecord/playback dilemma, I would suggest letting the user decide if he would like to use recording (mic, speaker) or not with a configuration.

[janoskranczler]

I've updated my answer above.

[std-s]

@janoskranczler

Hi, Is there a solution?

[cloudwebrtc]

please try this branch https://github.com/livekit/client-sdk-flutter/pull/312

[janoskranczler]

@cloudwebrtc I've seen that you merged the branch and I did try it but the speaker/earpiece change is still not working if the meeting is started with my microphone muted. If I unmute my microphone before the meeting starts then I can still mute it back and everything is working fine.

[davidzhao]

@janoskranczler do you mean that you've published a muted track?

Is this behavior reproducible with the included example app?

[janoskranczler]

@davidzhao Yes, I can reproduce with the example app if I mute my microphone I can't switch between speaker and earpiece.