Haoquan/rtp video stream control

[hq286]

Built a new remote python controller with low latency RTP video stream for mobile network connection. New python controller supports autopilot, recording, without crashing remote video stream.

[izivkov]

I am trying to run the new branch. One thing I noticed running the main app:

• after setting the Stream Mode to "RTP" in settings, and then selecting "Free Roam" fragment, the program crashes:

D/AndroidRuntime: Shutting down VM E/AndroidRuntime: FATAL EXCEPTION: main Process: org.openbot, PID: 26454 java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener at org.openbot.env.RtpServer.init(RtpServer.java:49) at org.openbot.env.PhoneController.init(PhoneController.java:64) at org.openbot.env.PhoneController.getInstance(PhoneController.java:33) at org.openbot.common.ControlsFragment.onViewCreated(ControlsFragment.java:76) at org.openbot.robot.FreeRoamFragment.onViewCreated(FreeRoamFragment.java:44) at androidx.fragment.app.Fragment.performViewCreated(Fragment.java:2987) at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:546) at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:282) at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:2189) at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:2106) at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:2002) at androidx.fragment.app.FragmentManager$5.run(FragmentManager.java:524) at android.os.Handler.handleCallback(Handler.java:938) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loop(Looper.java:250) at android.app.ActivityThread.main(ActivityThread.java:7755) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:958) V/FA: Recording user engagement, ms: 5450

[izivkov]

Is the new python script called "joystick_controller.py"? I cannot test it since I do not have a joystick. Is it a particular joystick model that works with it? I am wondering if the RTP stream would still be useful with keyboard-only controller, since most people do not have joystick.

[thias15]

In my opinion, having also a keyboard controller would be useful.

[thias15]

@haoquan on what OS did you test this? On Mac I'm unable to install approxeng.input. Is this essential or could it be done differently?

error: legacy-install-failure

[thias15]

Also, what kind of latency are you getting with RTP for the video stream?

[thias15]

@haoquan, after further testing I also encountered frequent app crashes.

java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener

Did you not face these app crashes?

[hq286]

I am trying to run the new branch. One thing I noticed running the main app:

• after setting the Stream Mode to "RTP" in settings, and then selecting "Free Roam" fragment, the program crashes:

D/AndroidRuntime: Shutting down VM E/AndroidRuntime: FATAL EXCEPTION: main Process: org.openbot, PID: 26454 java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener at org.openbot.env.RtpServer.init(RtpServer.java:49) at org.openbot.env.PhoneController.init(PhoneController.java:64) at org.openbot.env.PhoneController.getInstance(PhoneController.java:33) at org.openbot.common.ControlsFragment.onViewCreated(ControlsFragment.java:76) at org.openbot.robot.FreeRoamFragment.onViewCreated(FreeRoamFragment.java:44) at androidx.fragment.app.Fragment.performViewCreated(Fragment.java:2987) at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:546) at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:282) at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:2189) at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:2106) at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:2002) at androidx.fragment.app.FragmentManager$5.run(FragmentManager.java:524) at android.os.Handler.handleCallback(Handler.java:938) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loop(Looper.java:250) at android.app.ActivityThread.main(ActivityThread.java:7755) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:958) V/FA: Recording user engagement, ms: 5450

I have fixed this issue. Check out the latest commit

[hq286]

@haoquan, after further testing I also encountered frequent app crashes.

java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener

Did you not face these app crashes?

I fixed the issue. Check out the latest commit

[hq286]

Is the new python script called "joystick_controller.py"? I cannot test it since I do not have a joystick. Is it a particular joystick model that works with it? I am wondering if the RTP stream would still be useful with keyboard-only controller, since most people do not have joystick.

Yes. It should work with major bluetooth joystick controllers like PS4, and xbox and etc. I agree that a keyboard controller would be very convenient. The problem now is that I am using cv2 as the video displayer, which will cause conflict between keyboard control and cv2 video display (waitkey). It turns out that PyGame is introducing a lot of latency when displaying the video. I will think about how to resolve this issue.

[hq286]

@haoquan on what OS did you test this? On Mac I'm unable to install approxeng.input. Is this essential or could it be done differently?

error: legacy-install-failure

I developed and tested everything on a Linux computer. Unfortunately approxeng.input only supports Linux system. I'm stilly working on finding a solution for Mac.

[hq286]

Also, what kind of latency are you getting with RTP for the video stream?

I tested it on a Linux computer. The latency is about 150ms on average. It could vary depending on your network condition.

[izivkov]

That is pretty good response time. I will check it out. Can I run the app and see the video without a Joystick?

[izivkov]

I am trying to run the new branch. One thing I noticed running the main app:

• after setting the Stream Mode to "RTP" in settings, and then selecting "Free Roam" fragment, the program crashes:

D/AndroidRuntime: Shutting down VM E/AndroidRuntime: FATAL EXCEPTION: main Process: org.openbot, PID: 26454 java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener at org.openbot.env.RtpServer.init(RtpServer.java:49) at org.openbot.env.PhoneController.init(PhoneController.java:64) at org.openbot.env.PhoneController.getInstance(PhoneController.java:33) at org.openbot.common.ControlsFragment.onViewCreated(ControlsFragment.java:76) at org.openbot.robot.FreeRoamFragment.onViewCreated(FreeRoamFragment.java:44) at androidx.fragment.app.Fragment.performViewCreated(Fragment.java:2987) at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:546) at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:282) at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:2189) at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:2106) at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:2002) at androidx.fragment.app.FragmentManager$5.run(FragmentManager.java:524) at android.os.Handler.handleCallback(Handler.java:938) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loop(Looper.java:250) at android.app.ActivityThread.main(ActivityThread.java:7755) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:958) V/FA: Recording user engagement, ms: 5450

I have fixed this issue. Check out the latest commit

OK, the code works with RTP selected, but it crashes if we select WebRTC, and then select Free Roam fragment. RTSP does not crash.

[hq286]

Fixed it. There was a type with webrtc.

[hq286]

That is pretty good response time. I will check it out. Can I run the app and see the video without a Joystick?

Yes. You can run keyboard-pygame.py without the video flag. Then run the camera.py. Make sure you are in the RTP mode, and match the ports for control and stream

[hq286]

This works at least in Linux. I'm not sure about Mac. Getting GStreamer, opencv and python to work together on Mac needs some work

[thias15]

I compiled OpenCV with GStreamer support on Mac, but when I try to run the camera, I get the following error.

Failed to open camera
Src opened, 0x0 @ 0 fps

@izivkov did you manage to get this running under Linux?

[izivkov]

No, I got the same error. I never got video to work with the new Python script. I thought it was something wrong with my configuration.

On Wed., Dec. 28, 2022, 5:47 a.m. thias15, @.***> wrote:

I compiled OpenCV with GStreamer support on Mac, but when I try to run the camera, I get the following error.

Failed to open camera Src opened, 0x0 @ 0 fps

@izivkov https://github.com/izivkov did you manage to get this running under Linux?

— Reply to this email directly, view it on GitHub https://github.com/isl-org/OpenBot/pull/308#issuecomment-1366556823, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7M37RZMKFEDBNHXTSFSO3WPQLDNANCNFSM6AAAAAASJLNUYQ . You are receiving this because you were mentioned.Message ID: @.***>

[thias15]

@hq286 any idea what we are doing wrong?

[hq286]

No, I got the same error. I never got video to work with the new Python script. I thought it was something wrong with my configuration. … On Wed., Dec. 28, 2022, 5:47 a.m. thias15, @.> wrote: I compiled OpenCV with GStreamer support on Mac, but when I try to run the camera, I get the following error. Failed to open camera Src opened, 0x0 @ 0 fps @izivkov https://github.com/izivkov did you manage to get this running under Linux? — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7M37RZMKFEDBNHXTSFSO3WPQLDNANCNFSM6AAAAAASJLNUYQ . You are receiving this because you were mentioned.Message ID: @.>

It's the problem of GStreamer pipeline configuration. This configuration works in Ubuntu 20.04. I need to investigate the configuration on MacBook

[thias15]

No, I got the same error. I never got video to work with the new Python script. I thought it was something wrong with my configuration. … On Wed., Dec. 28, 2022, 5:47 a.m. thias15, @.> wrote: I compiled OpenCV with GStreamer support on Mac, but when I try to run the camera, I get the following error. Failed to open camera Src opened, 0x0 @ 0 fps @izivkov https://github.com/izivkov did you manage to get this running under Linux? — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7M37RZMKFEDBNHXTSFSO3WPQLDNANCNFSM6AAAAAASJLNUYQ . You are receiving this because you were mentioned.Message ID: @.>

How did you install OpenCV? Can you try the script I added to the repo? Make sure to install GStreamer before.

[izivkov]

@thias15 What is the name of your "install OpenCV" script and what branch is it in?

[thias15]

I pushed it to this PR, it's called install_opencv.sh

[thias15]

@izivkov any update on your end?

[thias15]

By the way, I think we should also unify the port setting, at least for the controller across the different scripts.

[izivkov]

@izivkov any update on your end?

I installed GStreamer and OpenCV using your installation script on my Linux PC but did not see video. Basically, the Video Window never appears. I used the following command:

python3 joystick_controller.py -video

Maybe it is my PC - only a Celaron CPU. I'm not sure if it needs a more powerful GPU.

Also, I'm not that familiar with GStreamer and OpenCV, so I may be doing some incorrectly.