client-sdk-js icon indicating copy to clipboard operation
client-sdk-js copied to clipboard

Published 20 hours ago • verified publisher icon livekit

AudioPlaybackStatus gets fired multiple times even after invoking startAudio()

Open karthiksirsikar opened this issue 2 years ago • 14 comments

Describe the bug

Im expecting audioPlaybackStatusChanged to not fire after calling room.startAudio() on user click. But audioPlaybackStatusChanged keeps firing on each new participant joined.

Reproduction

Happens in MS Edge Mobile on https://example.livekit.io/ whenever a new participant joins

Logs

chunk-vendors.d8055e62.js:316 room event {event: 'participantConnected', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'trackPublished', args: Array(2)}
chunk-vendors.d8055e62.js:316 room event {event: 'connectionQualityChanged', args: Array(2)}
chunk-vendors.d8055e62.js:316 received server offer {RTCSdpType: 'offer', signalingState: 'stable'}
chunk-vendors.d8055e62.js:316 room event {event: 'trackSubscriptionPermissionChanged', args: Array(3)}
chunk-vendors.d8055e62.js:316 room event {event: 'trackSubscribed', args: Array(3)}
app.a2ae327e.js:1 trackSubscribed Ba {_events: {…}, _eventsCount: 12, _maxListeners: undefined, audioLevel: 0, isSpeaking: false, …} Ha {_events: {…}, _eventsCount: 7, _maxListeners: undefined, metadataMuted: false, handleMuted: ƒ, …}
chunk-vendors.d8055e62.js:316 could not playback audio DOMException: play() can only be initiated by a user gesture.
Ns.handleAudioPlaybackFailed @ chunk-vendors.d8055e62.js:316
Or.emit @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
Promise.catch (async)
attach @ chunk-vendors.d8055e62.js:316
attach @ chunk-vendors.d8055e62.js:316
mounted @ app.a2ae327e.js:1
In @ chunk-vendors.d8055e62.js:63
Ni @ chunk-vendors.d8055e62.js:63
insert @ chunk-vendors.d8055e62.js:63
O @ chunk-vendors.d8055e62.js:63
(anonymous) @ chunk-vendors.d8055e62.js:63
Oi.e._update @ chunk-vendors.d8055e62.js:63
i @ chunk-vendors.d8055e62.js:63
e.get @ chunk-vendors.d8055e62.js:63
e.run @ chunk-vendors.d8055e62.js:63
Gi @ chunk-vendors.d8055e62.js:63
(anonymous) @ chunk-vendors.d8055e62.js:63
Vn @ chunk-vendors.d8055e62.js:63
Promise.then (async)
Hn @ chunk-vendors.d8055e62.js:63
Kn @ chunk-vendors.d8055e62.js:63
Xi @ chunk-vendors.d8055e62.js:63
e.update @ chunk-vendors.d8055e62.js:63
e.notify @ chunk-vendors.d8055e62.js:63
set @ chunk-vendors.d8055e62.js:63
(anonymous) @ app.a2ae327e.js:1
u @ chunk-vendors.d8055e62.js:216
(anonymous) @ chunk-vendors.d8055e62.js:216
(anonymous) @ chunk-vendors.d8055e62.js:216
a @ chunk-vendors.d8055e62.js:79
s @ chunk-vendors.d8055e62.js:79
(anonymous) @ chunk-vendors.d8055e62.js:79
t @ chunk-vendors.d8055e62.js:121
(anonymous) @ chunk-vendors.d8055e62.js:79
t @ app.a2ae327e.js:1
(anonymous) @ app.a2ae327e.js:1
r.emitEvent @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
Or.emit @ chunk-vendors.d8055e62.js:316
emit @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
Or.emit @ chunk-vendors.d8055e62.js:316
emit @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
Or.emit @ chunk-vendors.d8055e62.js:316
setTrack @ chunk-vendors.d8055e62.js:316
addSubscribedMediaTrack @ chunk-vendors.d8055e62.js:316
onTrackAdded @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
Or.emit @ chunk-vendors.d8055e62.js:316
cr.subscriber.pc.ontrack @ chunk-vendors.d8055e62.js:316
a @ chunk-vendors.d8055e62.js:316
a @ chunk-vendors.d8055e62.js:316
Show 23 more frames
chunk-vendors.d8055e62.js:316 room event {event: 'audioPlaybackChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 sending answer
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'audioPlaybackChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'trackMuted', args: Array(2)}
chunk-vendors.d8055e62.js:316 room event {event: 'participantConnected', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'trackPublished', args: Array(2)}
chunk-vendors.d8055e62.js:316 room event {event: 'connectionQualityChanged', args: Array(2)}
chunk-vendors.d8055e62.js:316 received server offer {RTCSdpType: 'offer', signalingState: 'stable'}
chunk-vendors.d8055e62.js:316 room event {event: 'trackSubscriptionPermissionChanged', args: Array(3)}
chunk-vendors.d8055e62.js:316 room event {event: 'trackSubscribed', args: Array(3)}
app.a2ae327e.js:1 trackSubscribed Ba {_events: {…}, _eventsCount: 12, _maxListeners: undefined, audioLevel: 0, isSpeaking: false, …} Ha {_events: {…}, _eventsCount: 7, _maxListeners: undefined, metadataMuted: false, handleMuted: ƒ, …}
chunk-vendors.d8055e62.js:316 could not playback audio DOMException: play() can only be initiated by a user gesture.
Ns.handleAudioPlaybackFailed @ chunk-vendors.d8055e62.js:316
Or.emit @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
Promise.catch (async)
attach @ chunk-vendors.d8055e62.js:316
attach @ chunk-vendors.d8055e62.js:316
mounted @ app.a2ae327e.js:1
In @ chunk-vendors.d8055e62.js:63
Ni @ chunk-vendors.d8055e62.js:63
insert @ chunk-vendors.d8055e62.js:63
O @ chunk-vendors.d8055e62.js:63
(anonymous) @ chunk-vendors.d8055e62.js:63
Oi.e._update @ chunk-vendors.d8055e62.js:63
i @ chunk-vendors.d8055e62.js:63
e.get @ chunk-vendors.d8055e62.js:63
e.run @ chunk-vendors.d8055e62.js:63
Gi @ chunk-vendors.d8055e62.js:63
(anonymous) @ chunk-vendors.d8055e62.js:63
Vn @ chunk-vendors.d8055e62.js:63
Promise.then (async)
Hn @ chunk-vendors.d8055e62.js:63
Kn @ chunk-vendors.d8055e62.js:63
Xi @ chunk-vendors.d8055e62.js:63
e.update @ chunk-vendors.d8055e62.js:63
e.notify @ chunk-vendors.d8055e62.js:63
set @ chunk-vendors.d8055e62.js:63
(anonymous) @ app.a2ae327e.js:1
u @ chunk-vendors.d8055e62.js:216
(anonymous) @ chunk-vendors.d8055e62.js:216
(anonymous) @ chunk-vendors.d8055e62.js:216
a @ chunk-vendors.d8055e62.js:79
s @ chunk-vendors.d8055e62.js:79
(anonymous) @ chunk-vendors.d8055e62.js:79
t @ chunk-vendors.d8055e62.js:121
(anonymous) @ chunk-vendors.d8055e62.js:79
t @ app.a2ae327e.js:1
(anonymous) @ app.a2ae327e.js:1
r.emitEvent @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
Or.emit @ chunk-vendors.d8055e62.js:316
emit @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
Or.emit @ chunk-vendors.d8055e62.js:316
emit @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
Or.emit @ chunk-vendors.d8055e62.js:316
setTrack @ chunk-vendors.d8055e62.js:316
addSubscribedMediaTrack @ chunk-vendors.d8055e62.js:316
onTrackAdded @ chunk-vendors.d8055e62.js:316
(anonymous) @ chunk-vendors.d8055e62.js:316
Or.emit @ chunk-vendors.d8055e62.js:316
cr.subscriber.pc.ontrack @ chunk-vendors.d8055e62.js:316
a @ chunk-vendors.d8055e62.js:316
a @ chunk-vendors.d8055e62.js:316
Show 23 more frames
chunk-vendors.d8055e62.js:316 room event {event: 'audioPlaybackChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 sending answer
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'audioPlaybackChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}
chunk-vendors.d8055e62.js:316 room event {event: 'activeSpeakersChanged', args: Array(1)}

System Info

Android 12, Microsoft edge

Severity

blocking all usage of LiveKit

Additional Information

No response

karthiksirsikar avatar Sep 28 '22 06:09 karthiksirsikar

MS Edge on mobile seem to have strict policy about when an audio element can be autoplayed. From what I can gather, any new audio element that's created after the user has given autoplay consent will need to be granted explicitly again.

The following case does not require reconsent:

  • after approval, existing user leaves and rejoins
  • after approval, existing user leaves and another joins

These cases work because we recycle audio elements for this reason.

The following case does require reconsent:

  • after approval, an additional user joins (this involves a new audio element to be created)

Since this is an Edge security feature, I'm unsure if there's anything we can do about it.

davidzhao avatar Oct 06 '22 07:10 davidzhao

maybe the work in https://github.com/livekit/client-sdk-js/pull/446 could serve as a workaround for this issue on edge, although not sure if that makes a difference at all if the autoplay is blocked on the element level.

lukasIO avatar Oct 06 '22 10:10 lukasIO

that's a good idea! @karthikSirsikar Do you want to give this a try?

  • run against pre-release main
  • pass this option in when creating the room
expWebAudioMix: true

davidzhao avatar Oct 07 '22 16:10 davidzhao

Yes, this seems to work on MS Edge android but not on IOS Safari

karthiksirsikar avatar Oct 10 '22 08:10 karthiksirsikar

on which iOS version do you see this failing? (and is there any error being thrown?)

lukasIO avatar Oct 10 '22 08:10 lukasIO

15.4.1 on IPad

karthiksirsikar avatar Oct 10 '22 09:10 karthiksirsikar

This is the error: NotAllowedError: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission

karthiksirsikar avatar Oct 10 '22 09:10 karthiksirsikar

this sounds like you didn't allow the website to use microphone/camera! make sure to test it on a secure host, it probably won't work for iOS if you use http instead of https

lukasIO avatar Oct 10 '22 09:10 lukasIO

@karthikSirsikar does it work for you ? just tried to reproduce the issue you mentioned (before deleting) when not publishing, but couldn't reproduce it

lukasIO avatar Oct 10 '22 13:10 lukasIO

Sorry, that was misinterpreted. I had disabled expWebAudioMix for IOS and had only enabled it for android devices.

karthiksirsikar avatar Oct 10 '22 13:10 karthiksirsikar

But now the issue is on android, the audio is going into loop. Should i skip attaching track to audio tag when expWebAudioMix is enabled

karthiksirsikar avatar Oct 10 '22 13:10 karthiksirsikar

hm. it shouldn't create a feedback loop when calling attach. does the feedback loop also happen when running the sample ? (the sample also uses attach)

lukasIO avatar Oct 10 '22 13:10 lukasIO

i meant the one in this repo, started with yarn sample

lukasIO avatar Oct 10 '22 14:10 lukasIO

Yes feedback loop is happening. I enabled expWebAudioMix in the sample and tried

karthiksirsikar avatar Oct 11 '22 11:10 karthiksirsikar

Coming back to this: Could it be that what you were seeing was the fact that echoCancellation doesn't work with webAudio? Or worded differently: does the loopback also happen when using headphones on the phone?

lukasIO avatar Oct 28 '22 12:10 lukasIO

for reference, this is the chromium bug (the issue mentions Android explicitly as well) https://bugs.chromium.org/p/chromium/issues/detail?id=687574#c60 sounds like the fix will be enabled by default soon in chrome!

lukasIO avatar Oct 28 '22 15:10 lukasIO

expWebAudioMix does seem to solve the original issue for edge, feel free to open a new issue for the feedback loop in case this is still happening.

lukasIO avatar Nov 30 '22 09:11 lukasIO