Add JVM Support

[aschulz90]

I only tested this on a Windows PC connecting with an Android device. Video/Screensharing and Audio works with the sample apps. DataChannel was only tested via Junit. Since this relies on webrtc-java for native interop, it should theoretically work on any platform it supports as well.

[shepeliev]

@aschulz90 great job. Let me take some time to review.

[shepeliev]

I've tried to run the JVM sample app on Mac. Unfortunately, there is exception getting user media:

java.lang.Error: Unhandled Exception
        at dev.onvoid.webrtc.media.video.VideoDeviceSource.start(Native Method)
        at com.shepeliev.webrtckmp.LocalVideoStreamTrack.<init>(LocalVideoStreamTrack.kt:13)
        at com.shepeliev.webrtckmp.MediaDevicesImpl.getUserMedia(MediaDevices.kt:89)
        at com.shepeliev.webrtckmp.MediaDevices$Companion.getUserMedia(MediaDevices.kt)
        at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel$openUserMedia$2.invokeSuspend(RoomComponent.kt:86)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

Also, JVM tests fail on Mac with java.lang.UnsatisfiedLinkError yet. I'll try to figure out what is the reason as soon as be able.

It's really great to add JVM platform, however I think we should make it work properly on any OS (windows, macos, linux).

[tamimattafi]

@shepeliev @aschulz90 Any updates on this?

[aschulz90]

@tamimattafi I don't have any Mac (neither Intel nor M1) or Linux device with a webcam and microphone to test on. Maybe you could help by checking the tests and demo on them, if you have any of those device types.

[tamimattafi]

@aschulz90 Hello! I have both Mac on Intel and M1, I can help on Saturday if it's suitable for you

[tamimattafi]

@aschulz90 We tried on both M1 and Intel macs, here's a report:

1. Sample run from main():

1. Crash when the PC on Intel chip is missing a camera or microphone:

java.util.NoSuchElementException: List is empty.
        at kotlin.collections.CollectionsKt___CollectionsKt.first(_Collections.kt:214)
        at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel.createPeerConnection(RoomComponent.kt:180)
        at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel.createRoom(RoomComponent.kt:122)

2. Selecting Speakers on M1 MacBook throws an exception:

java.lang.Error: Set playout device failed
	at dev.onvoid.webrtc.media.audio.AudioDeviceModule.setPlayoutDevice(Native Method)
	at com.shepeliev.webrtckmp.WebRtc.setAudioOutputDevice(WebRtc.kt:78)
	at com.shepeliev.webrtckmp.SelectMicrophoneCameraScreenKt$SelectMicrophoneCameraScreen$4$1$3$1$1.invoke(SelectMicrophoneCameraScreen.kt:166)
	at com.shepeliev.webrtckmp.SelectMicrophoneCameraScreenKt$SelectMicrophoneCameraScreen$4$1$3$1$1.invoke(SelectMicrophoneCameraScreen.kt:163)
	at com.shepeliev.webrtckmp.SelectMicrophoneCameraScreenKt$DeviceSelector$1$4$2$1.invoke(SelectMicrophoneCameraScreen.kt:235)
	at com.shepeliev.webrtckmp.SelectMicrophoneCameraScreenKt$DeviceSelector$1$4$2$1.invoke(SelectMicrophoneCameraScreen.kt:234)

3. The confirm button on M1 MacBook doesn't react to clicks, even tho all devices were selected, seems like it's missing a permission
4. Share desktop asks for permission, denying the permission, keeps showing the screen sharing, and by clicking on create (room) it throws this exception:

java.util.NoSuchElementException: List is empty.
	at kotlin.collections.CollectionsKt___CollectionsKt.first(_Collections.kt:214)
	at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel.createPeerConnection(RoomComponent.kt:179)
	at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel.createRoom(RoomComponent.kt:122)
	at com.shepeliev.webrtckmp.sample.shared.RoomComponent.createRoom(RoomComponent.kt)
	at com.shepeliev.webrtckmp.VideoScreenKt$VideoScreen$2$1$2$2$1$1.invoke(VideoScreen.kt:122)
	at com.shepeliev.webrtckmp.VideoScreenKt$VideoScreen$2$1$2$2$1$1.invoke(VideoScreen.kt:122)
	at androidx.compose.foundation.ClickablePointerInputNode$pointerInput$3.invoke-k-4lQ0M(Clickable.kt:898)

Because of these issues, I couldn't really test the actual connection. Not sure how to give this permission manually since I don't see the app there, probably I need to build a .dmg and install the app, or something is missing in the manifest / plist for macOS.

2. Tests in commonTest run on jvm:

1. IceServerTest fails both on M1 and Intel:

java.lang.UnsatisfiedLinkError: 'void dev.onvoid.webrtc.media.audio.AudioDeviceModule.initialize(dev.onvoid.webrtc.media.audio.AudioLayer)'
	at dev.onvoid.webrtc.media.audio.AudioDeviceModule.initialize(Native Method)
	at dev.onvoid.webrtc.media.audio.AudioDeviceModule.<init>(AudioDeviceModule.java:36)
	at com.shepeliev.webrtckmp.WebRtc.initializePeerConnectionFactory(WebRtc.kt:42)
	at com.shepeliev.webrtckmp.WebRtc.initialize(WebRtc.kt:37)
	at com.shepeliev.webrtckmp.WebRtc.getPeerConnectionFactory$webrtc_kmp(WebRtc.kt:16)
	at com.shepeliev.webrtckmp.PeerConnection$native$2.invoke(PeerConnection.kt:40)
	at com.shepeliev.webrtckmp.PeerConnection$native$2.invoke(PeerConnection.kt:39)
	....

2. PeerConnectionTest fails both on M1 and Intel:

java.lang.UnsatisfiedLinkError: 'void dev.onvoid.webrtc.media.audio.AudioDeviceModule.initialize(dev.onvoid.webrtc.media.audio.AudioLayer)'
	at dev.onvoid.webrtc.media.audio.AudioDeviceModule.initialize(Native Method)
	at dev.onvoid.webrtc.media.audio.AudioDeviceModule.<init>(AudioDeviceModule.java:36)
	at com.shepeliev.webrtckmp.WebRtc.initializePeerConnectionFactory(WebRtc.kt:42)
	at com.shepeliev.webrtckmp.WebRtc.initialize(WebRtc.kt:37)
	at com.shepeliev.webrtckmp.WebRtc.getPeerConnectionFactory$webrtc_kmp(WebRtc.kt:16)
	at com.shepeliev.webrtckmp.PeerConnection$native$2.invoke(PeerConnection.kt:40)
	at com.shepeliev.webrtckmp.PeerConnection$native$2.invoke(PeerConnection.kt:39)
	...

@shepeliev @aschulz90 I would love to help with any further steps to accelerate the integration on JVM

[shepeliev]

@tamimattafi thanks for the tests. Unfortunately, I don't have any quick answer for these issues on mac. It definitely requires more researching. I'll back to this as soon as be able.

[aschulz90]

Yeah, I haven't accounted for permissions on desktop. A quick search wasn't that conclusive on how to request them for java applications on macOS. Unfortunately I don't think I can be of much help in that area.

[tricknology]

@tamimattafi - Are you using OpenJDK by any chance? And perhaps running this from the IDE and not a built package?

Check this: https://bugs.openjdk.org/browse/JDK-8272639

They've used the --resource-dir option when building the package and pointed it to a custom info.plist containing the proper key/val

<key>NSMicrophoneUsageDescription</key>
<string>The application is requesting access to the microphone.</string> 

[tamimattafi]

@tricknology Yes indeed, I'm using OpenJDK 20 on my machine, and JetBrains Runtime version 17 on the IDE. And indeed I run the sample from main() method from the IDE as stated in the test report.

My guess is, as you stated that it is better to build a package and install it properly for proper testing with permissions.

[its-Chiedu]

@tamimattafi I feel like there should be a way to pass options to the compiler in run config.

You might also specify a makefile to do the same.. unfortunately I'm not exactly sure on the step-by-step.

[aschulz90]

Selecting Speakers on M1 MacBook throws an exception:

This should be fixed now.

[tamimattafi]

@its-Chiedu There is a way to pass every detail including info.plist https://github.com/JetBrains/compose-multiplatform/blob/master/tutorials/Native_distributions_and_local_execution/README.md#customizing-infoplist-on-macos

However, I'm having some issues building a .dmg using java 17 and java 20, I will look through the issue when I find some free time.

[aschulz90]

@shepeliev I have merged the latest changes from main and updated the sample app for jvm support. Can you please review that part again?

[shepeliev]

Unfortunately, still have an exception on both of Intel and M2 macs:

java.lang.Error: Unhandled Exception
        at dev.onvoid.webrtc.media.video.VideoDeviceSource.start(Native Method)
        at com.shepeliev.webrtckmp.LocalVideoStreamTrack.<init>(LocalVideoStreamTrack.kt:13)
        at com.shepeliev.webrtckmp.MediaDevicesImpl.getUserMedia(MediaDevices.kt:89)
        at com.shepeliev.webrtckmp.MediaDevices$Companion.getUserMedia(MediaDevices.kt)
        at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel$openUserMedia$2.invokeSuspend(RoomComponent.kt:86)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

I guess some problem is in native WebRTC SDK that embeded with WebRTC java lib. There are two options for now:

• make basic sample Java app which reproduces the problem and raise an issue at https://github.com/devopvoid/webrtc-java
• build webrtc-java from the sources adding some logs into low-level C code and try to figure out by myself

I think we could start with the first option for now. Java background devs help is appreciated :)