Feat. Request: Share audio

[igorcafe]

Meet has this feature when you share a specific tab on Linux, and I believe you can share output sound on Windows too.

[muzam1l]

So yeah, audio in screen sharing was not set, can you test this to see if it meets your needs.

https://deploy-preview-5--mooz-app.netlify.app

[igorcafe]

I tried it here, but it doesn't seem to work. I am using Chrome latest on Linux. A feature that works in Meet is sharing a specific tab with sound enabled, but that didn't work on mooz.

[muzam1l]

I am on Windows, and this deploy preview shows option for Tab audio like this.

Can you share the screenshots or additional details, how it shows on your machine.

[resetcoder]

I had the same issue and I found that this clause in /src/utils/hooks/use-streams.tsx didn't have the attribute for audio, so I changed it to:

            const stream = await (navigator.mediaDevices as any).getDisplayMedia({
                video: { cursor: 'always' },
                audio: true
            })

Now I have the "Share system audio" or "Share tab audio" checkboxes on the sharing window, but I still don't think audiostream goes through.

I'm not familiar with this code, but as I see screen/window/tab sharing is used as a media device and if I add the audio: true attribute then it should attach audio to the stream object. The only I can think of is somewhere where this stream object is processed the audio track is simply isn't attached to the routing to the clients. Any idea?

[muzam1l]

Yep, now I remember, screen share audio track is "unhandled" by design.

Basically managing tracks in peer connection is kinda ugly here, so how I go about it is removing audio track entirely if you say mute and add it back if you unmute instead of just, you know, muting and unmuting it (enabled fields to be precise) :).

Correctly handling more than one audio or more than one video gets ugly in peer to peer. Getting even addition video track is handled trickily, all dirty code in this file. Therefore stream audio is not here for now!

[resetcoder]

But I just don't get it, if the screen capture is just a stream, then why not just distribute it as is to all the clients, and if you already showing the stream in a video player element at client side, it can play both the video and audio for everyone. If the client who shares the screen doesn't tick the checkbox "share audio", then the stream simply won't have audio track, therefore will be mute.

I think the audio sharing problem could be resolved by not removing the audio track from the stream, and the client who is sharing can decide if audio is shared or not.

I'm totally not familiar how these things work, tell me if I'm wrong.

[muzam1l]

Yes you can send the multiple tracks of audio and video in peer connection. However they do not have deterministic id or anyway to differentiate between two audio and video tracks, so no information for even cleaning up some old "dead" tracks. That's why I send new track instead of muting-unmuting the existing track so other client can just destroy previous tracks and use new ones, that's unfortunate and getting even addition video track is just hacky.

Well additional audio can be made to work with similar hacks even in current setup, but it's just peer connection api is so shitty that I don't want to deal with it just now.

PS: It can be made to work just fine initially, but making it resistant to stand in all cases of scenarios, like mute-unmute, network errors, multiple peers, etc is tough.

Edit: It might not all make sence, it's just that I have dealt extensively with peer api and it sucks!

[resetcoder]

I see. I'm not familiar with this kind of js, I don't know how the peer handling works, I thought you handle each data packet yourself, that could make things much simpler, if you can just add a metadata to each packet what is what. If it is a built-in stuff, then I understand you have to "hack" your solution into it, that's not ideal.

I wanted to use this application to a web meeting and it works very well, except the audio sharing. I had to do a workaround to mix the desktop audio with the microphone in a virtual audio device and use it as my microphone, that worked.

So I understand the issue now and give up hacking in the audio sharing. Thanks anyways! Nice little app, though.

[muzam1l]

That's the issue, you can send metadata as seprate data channel, but media tracks do not even have same Id across connection, so you don't have any information on recieving tracks.

And glad that you liked it, thanks.

[muzam1l]

A little update on this. I rewrote the whole app logic and many things should make sense now! So Tab audio can now be shared wherever the browser allows it in https://github.com/muzam1l/mooz/pull/16.

You can check the preview at here. There may still be bugs on that branch as it is still in early development and not thoroughly tested and many things need improvements. But it is still a huge step up from the last version.