OpenGL GetProcAddress function

[bl1nch]

Hi, Kivy developers! I decided to try using the MPV video player as a backend for video rendering. There are python-mpv ctypes bindings for the MPV core library https://github.com/jaseg/python-mpv, that supports video rendering directly into OpenGL teture. There is an example in its documentation that shows how to use it with OpenGl. https://github.com/dfaker/imgui_glfw_pythonmpv_demo/blob/main/main.py

But in this example MPV requires GetProcAndress OpenGL function that I can't find in Kivy graphics instructions.

def get_process_address(_, name):
    print(name)
    address = glfw.get_proc_address(name.decode('utf8'))
    return ctypes.cast(address, ctypes.c_void_p).value

As I understand Kivy based on OpenGL, so I would like to know if it possible to implement this function to support MPV integration, because as I understand it, now video rendering in Kivy implemented not the best way. I tried to figure out how it works, and as I understand it, decoded by ffmpeg frames copied from video memory to RAM and then rendered by Kivy, that gives a high CPU load. So it seems like the MPV can solve it.

Thanks in advance for your reply!

[Julian-O]

I interpret this as a new feature request: "Expose OpenGL's get_proc_address() through kivy.graphics.opengl and/or `kivy.graphics.gl_instructions'"

(No comment on whether this is a good idea; I am out of my depth on that.)

[bl1nch]

Hi, is there any news on this issue?

[Julian-O]

Hi @bl1nch:

I think we might need to reset expectations here.

This is an Open Source project based on volunteers.

This request has been promptly added to the long list of new feature suggestions, where it will sit - hopefully garnering upvotes from people who agree it is useful - until a volunteer decides that they want to work for what I predict will be dozens of hours implementing (perhaps hundreds given the number of platforms to consider), and other volunteers can review it.

At the moment, there doesn't seem to be a strong justification: making it eventually possible to use yet another video player, because you don't like the current implementations, for unclear reasons.

Now, maybe other developers will know more about this and see the value in spending that time, but I see a lot of other higher priority suggestions, and would predict that this may take months or more likely years before it is attempted.

If that doesn't meet your needs, you need to provide your own resources to get it implemented - implement the change yourself, or persuade others to do so. (At the moment, there is no bounty system available here, or I would suggest that.)

[bl1nch]

Hi all. I'd like to leave some information here if anyone ever decides to implement this integration. I've never worked with OpenGL directly and I'm unlikely to be able to solve the problems described, but maybe for someone who decides to do this, this information will save some time, or maybe someone can explain where I went wrong.

Since kivy uses SDL window I thought to use the SDL_GL_GetProcAddress function. I edited kivy/core/window/_window_sdl2.pyx

def get_proc_address(self, name):
    cdef char *c_name = name
    cdef void* p = SDL_GL_GetProcAddress(c_name)
    return (<int*>p)[0]

Next I modified kivy/core/window/window_sdl2.py

def get_proc_address(self, name):
    return self._win.get_proc_address(name)

When I use this in kivy if I pass the correct OpenGL name to the function it works fine and the function returns the address.

from kivy.core.window import Window

address = Window.get_proc_address(b'glGetString')

But when I use this in MPV I get the error:

from ctypes import cast, c_void_p
from kivy.core.window import Window
from mpv import MPV, MpvRenderContext, MpvGlGetProcAddressFn

def gl_get_proc_address(_, name):
    address = Window.get_proc_address(name)
    return cast(address, c_void_p).value

class Example():
    def __init__(self):
        self._mpv = MPV(log_handler=print, loglevel='debug', ytdl=True)
        self._proc_addr_wrapper = MpvGlGetProcAddressFn(gl_get_proc_address)
        self._ctx = MpvRenderContext(
            self._mpv, 'opengl',
            opengl_init_params={'get_proc_address': self._proc_addr_wrapper}
        )

File "C:\Users\bl1nc\PycharmProjects\test\providers\mpv\__init__.py", line 14, in __init__
self._ctx = MpvRenderContext(
           ^^^^^^^^^^^^^^^^^
File "C:\Users\bl1nc\PycharmProjects\test\venv\Lib\site-packages\mpv.py", line 2058, in __init__
_mpv_render_context_create(buf, mpv.handle, kwargs_to_render_param_array(kwargs))
OSError: exception: access violation writing 0xFFFFFFFF83485340

I tried using pysdl2 to use a ctypes wrapper over SDL2.dll

from ctypes import cast, c_void_p
from sdl2 import SDL_GL_GetProcAddress

def gl_get_proc_address(_, name):
    address = SDL_GL_GetProcAddress(name)
    return cast(address, c_void_p).value

Then MPV successfully creates MpvRenderContext, but when rendering I get:

v vo/libmpv mpv_render_context_render() not being called or stuck.

Also MPV requires FBO id for rendering

def play(self):
    self._mpv.play(self.source)
    self._ctx.render(flip_y=False, opengl_fbo={'w': int(self.width), 'h': int(self.height), 'fbo': self.fbo.gl_id})

So I thought to modify kivy/graphics/fbo.pyx

@property
def gl_id(self):
    return self.buffer_id

But I'm not sure that this will work, since I did not solve the problem with MPV initialization and could not check it.

This is actually a very important issue. I encountered this problem because I have a ready-made application working with video streams, written in Kivy. Using this application on weak devices I get severe video stuttering. Also, ffpyplayer does not handle video stream errors well and the application crashes if errors are encountered in the stream. In my application I also use VLC player, but rendering in it works on the same principle as in ffpyplayer. Also, for example, Flutter currently has the same problem. https://github.com/alexmercerind/dart_vlc/issues/345

I will be very grateful if anyone can help with the problem described. Thank you in advance.

[bl1nch]

Hi all! I have some updates on this issue.

1. This error occurs due to the fact that the function ctx.render() must be called constantly, taking into account the FPS of the video

v vo/libmpv mpv_render_context_render() not being called or stuck.

2. This function returned the wrong address due to an incorrect cast of void pointer to int. This is the correct cast:

from libc.stdint cimport intptr_t

def get_proc_address(self, name):
    cdef char *c_name = name
    cdef void * p = SDL_GL_GetProcAddress(c_name)
    return <intptr_t>p

3. gl_id property works correctly

Taking all this into account, I made such a widget to test the MPV player, and I even managed to display the video in the Kivy window. But because of this, the entire interface of my application turned into just a black picture. If anyone has any ideas why this is happening please write, I would be grateful for any help!

from ctypes import cast, c_void_p
from mpv import MPV, MpvRenderContext, MpvGlGetProcAddressFn
from kivy.clock import Clock
from kivy.graphics.fbo import Fbo
from kivy.core.window import Window
from kivy.uix.image import Image


def gl_get_proc_address(_, name):
    address = Window.get_proc_address(name)
    return cast(address, c_void_p).value


class FboTest(Image):
    def __init__(self, **kwargs):
        self.fbo = Fbo(size=(Window.width, Window.height))
        self.mpv = MPV(log_handler=print, loglevel='debug', ytdl=True)
        self.proc_addr_wrapper = MpvGlGetProcAddressFn(gl_get_proc_address)
        self.ctx = MpvRenderContext(
            self.mpv, 'opengl',
            opengl_init_params={'get_proc_address': self.proc_addr_wrapper}
        )
        super(FboTest, self).__init__(**kwargs)
        self.play()

    def play(self):
        self.mpv.play("https://cam.kt.kg/cam14/stream.m3u8")
        Clock.schedule_interval(self.update, 1.0 / 25)

    def update(self, _):
        self.ctx.render(flip_y=True,
                        opengl_fbo={'w': int(Window.width), 'h': int(Window.height), 'fbo': self.fbo.gl_id})
        self.texture = self.fbo.texture
        self.canvas.ask_update()

https://github.com/kivy/kivy/assets/130155870/a4c074d2-dbbe-4742-880b-20bc55c27805

[Julian-O]

I have nothing useful to contribute about the bug, sorry. But I am excitedly cheering your attempts at tackling this! Good luck!

[Samael-TLB]

Possibly during the phase when the gl context passes to mpv kivy gl context is being corrupted kindly try to look what does actually mpv actually do under the hood before returning the context.

[bl1nch]

The problem solved by calling from kivy.graphics.instructions cimport reset_gl_context after each MPV render call. (Thanks to @bi0noid from the Kivy discord server for the tip!)

[Benexl]

If the problem is fixed could it be added to the kivy main library as a video provider because mpv handles video streaming pretty well and you can also easily add subtitles to the video you're streaming.

[bl1nch]

If the problem is fixed could it be added to the kivy main library as a video provider because mpv handles video streaming pretty well and you can also easily add subtitles to the video you're streaming.

The problem is not fixed yet. There are a lot of problems with context, rendering thread and other... You can read more about it here