Basic Computer Vision extension for getting frames from videos at YouTube

[jmplaza]

[jessiffmm]

We're having trouble with the implementation. As a first step, we wanted to play the YouTube video in the same place where the camera feed is displayed. For the time being we had replaced the use of Camera with VideoYoutube. But we get the error: Invalid hook call. Hooks can only be called inside of the body of a function component. This could happen for one of the following reasons:

1. You might have mismatching versions of React and the renderer (such as React DOM)
2. You might be breaking the Rules of Hooks
3. You might have more than one copy of React in the same app

We did an example in the folder prueba_youtube. This can be executed by putting: npx serve . And then opening the page at the address http://localhost:3000

[codezerro]

@jessiffmm Provide the folder/file/branch name.

[jessiffmm]

@codezerro The branch is https://github.com/JdeRobot/RoboticsAcademy/tree/issue-3058

[belal-bh]

Hi, is there anyone working on this issue or it has already been fixed ? I am interested to work on this issue.

[jmplaza]

HI @belal-bh , Yes some contributors are working on it. But it is a difficult issue, feel free to explore your own solution. And even share here your progress, or interact in this thread with those contributors. Take a look at the branch reported by @jessiffmm . There is no need to explicit assignment of the issue.

[codezerro]

Hi @belal-bh,

Thanks for your interest. We're now working on a feature where users can upload/attach a local video on the browser to run a computer vision exercise.

[Thakor-Yashpal]

Instead of directly replacing the camera component with a YouTube component, you should implement a single, flexible VideoPlayer component. This component will be responsible for deciding which video source (camera, local file, or YouTube) to render based on a simple prop, making the integration clean and less error-prone.

[jmplaza]

Hi @Thakor-Yashpal ,

thanks for your feedback. Where should that videoplayer run? In case of being the browser it would be great. Do you have a PR for that?

Real camera support is implemented with WebRTC technology inside the browser, which allows its flawless operation in Linux, Windows and MacOS machines. The "camera driver" runs inside the browser and sends frames to the robotics application running inside the RoboticsBackend (=docker image).

Local file and local video support is implemented locally inside the RoboticsBackend (RB) which is sandboxed (it should not have internet access for security reasons).

[Thakor-Yashpal]

Hi @jmplaza,

Thanks for the detailed clarification!

My suggestion for the FlexibleVideoPlayer is that it would run entirely in the browser. Here's how it would handle the different sources:

• For Camera: It would use WebRTC, just as your current implementation does.
• For YouTube: It would handle the YouTube URL (e.g., using a library like react-player).
• For Local File: It would use a standard browser <input type="file"> element. The user would manually select a file, and the component could then use URL.createObjectURL() to get a secure, temporary, in-memory URL to play the video directly in the browser's <video> tag.

The browser completely sandboxed this approach, eliminating security risks, as the website never gains direct access to the user's file system. This would unify all video-handling logic on the client side and remove the need to send local files to the RoboticAttackend for processing.

Let me know your thoughts!

[jmplaza]

Hi @Thakor-Yashpal ,

the use of react-player (https://www.npmjs.com/package/react-player) sounds really good to me. As long as we can get periodic frames in the browser from the YouTube (and vimeo...) video player we could send them to the robotics application running in the RoboticsBackend in the same way we do for real cameras (the frames are obtained from the webRTC sw in that case).

Are you up for doing a prototype? Just a single static webpage prototype (a) getting one frame from react-player sw (and a demo YouTube URL) and (b) showing that at a that static webpage. In order to reduce complexity for now, not integrated in RoboticsAcademy frontend yet. That initial prototype could be improved later for getting+sending not only a single frame, but many of them in loop, continuous, periodic way.

[Thakor-Yashpal]

Hi @implaza,

Thank you for laying out the plan. The idea of starting with a static webpage prototype using react-player to get the frames sounds like a great approach.

I'm happy to work on this. I'll be able to start on Friday and will post my progress here.