drake icon indicating copy to clipboard operation
drake copied to clipboard

Remove X server requirement from OpenGL and VTK renderers

Open pathammer opened this issue 1 year ago • 2 comments

Is your feature request related to a problem? Please describe. According to @jwnimmer-tri, there's no reason for an X server to be a strict requirement for these renderers. It makes it harder to deploy to headless environments for containerized environments.

Describe the solution you'd like RenderEngineGL could use EGL to create it's OpenGL context. VTK can automatically use EGL if built with that option.

Describe alternatives you've considered I considered using xvfb but that does non-HW-accelerated rendering and would be a bottleneck.

pathammer avatar Feb 26 '24 22:02 pathammer

FYI on a related topic...

As of Bazel 7.0, to access a real Xorg server during Bazel tests downstream projects must add this line in their .bazelrc:

build --sandbox_add_mount_pair=/tmp/.X11-unix

Otherwise, you'll see an failure such as DRAKE_THROW_UNLESS(display != nullptr); around internal_opengl_context.cc:85.

This is due to the following change in Bazel 7: --incompatible_sandbox_hermetic_tmp is enabled by default.

jwnimmer-tri avatar Mar 09 '24 17:03 jwnimmer-tri

I tried some prototyping today...

I used this WIP branch that switches VTK rendering to use EGL instead of GLX. I built a wheel file, installed it on Deepnote, and did apt install libegl-mesa0 instead of install_xvfb. I ran rendering_multibody_plant.ipynb and it worked fine. Thus, EGL seems like a viable option for CPU-based rendering without an X server.

As I understand it, there are ways to teach EGL to use the GPU, but I haven't looked into that specifically yet.

When using EGL, the camera option to show_window = True seemingly has no effect, at least not in my WIP branch. Possibly some device configuration is necessary, or maybe I need to be using Wayland instead of Xorg. Also, most of the VTK render engine tests pass, but two cases fail (for me).

Assuming that EGL can't be made strictly better than GLX, the next steps would be to see if we can compile both GLX and EGL support into Drake's VTK, and offer a RenderEngineVtkParams option for which one to use.

jwnimmer-tri avatar Mar 30 '24 17:03 jwnimmer-tri

For our Kitware helpers -- let's say the first milestone on this issue is just RenderEngineVtk. We'll leave the RenderEngineGl for a second milestone.

jwnimmer-tri avatar Jul 09 '24 15:07 jwnimmer-tri

Actually, I take that back. Let's use #21700 for the VTK option, and keep this ticket for the RenderEngineGl changes.

jwnimmer-tri avatar Jul 09 '24 21:07 jwnimmer-tri