Rewrite ui in Raylib (from Qt)

[adeebshihadeh]

openpilot's UI used to be written in https://github.com/memononen/nanovg, and in retrospect it was nicer to work in than Qt. Qt is nice for cabana, but it turns out to not add much value to the openpilot ui and adds a bunch of conceptual and dependency complexity.

Some options

• https://www.raylib.com/
• https://github.com/Immediate-Mode-UI/Nuklear
• https://github.com/raysan5/raygui

We're looking for:

• cross platform: Linux, macOS, etc.
• small, simple. single header is ideal
• Python bindings? would be great to have all or most of the UI written in Python

The onroad camera view is a good test for trying out these libraries.

[adeebshihadeh]

@deanlee want to try to make a small demo in raylib of the onroad camera view?

[deanlee]

I tried writing a simple demo based on Raylib to display the road camera (without shaders and transforms, just to quickly get a feel for Raylib's basic features). Raylib can simplify some OpenGL code, and its text rendering performance is also somewhat better (currently, in the drawHud function of the Qt UI, the three drawText function calls consumes about 3% of the CPU). Raylib might be a good option. The real challenge is the workload of migrating the entire UI to Raylib, including rewriting network management, WiFi manager, and multilingual support, among other features. Additionally, Raylib itself lacks a signal-slot mechanism. All logic is driven by a while loop, and input/output would also need to be restructured, making the workload quite significant. Considering that the current UI is relatively stable, if the goal is just to save space, it might be worth first optimizing other dependencies that can be relatively easily removed before eventually migrating to Raylib, RayUI, or another library.

Although it's possible to use Raylib within a Qt widget to render graphics, this actually increases complexity and maintenance difficulty. There's no good way to gradually transition to Raylib and remove the Qt dependency through multiple PRs. It looks like a complete rewrite would be necessary.

[adeebshihadeh]

Qt was a big improvement over the old APKs, and I expect this to be a similar improvement. Smaller dependency, simpler library, better docs, etc. I also expect it to help with https://github.com/commaai/openpilot/issues/30294.

I think we do have to find some way to do this incrementally; like you said, it's too much to do all at once. If you can't figure out a way to do it, I can start it. Want to put up a draft PR with the demo you wrote?

[deanlee]

I'll commit a draft after adding the shader, transformations, and other features to ensure it matches the CameraView's output.

[AskAlice]

fyne is quite nice although it's written in golang which would be a bit of a hurdle, though it wouldn't be the first time the UI were separate from the py. I think it also might be worth considering from this list https://github.com/sudhakar3697/awesome-electron-alternatives#python

[deanlee]

PR #33326 provides a draft on using Raylib to display the road camera, with automatic scaling to fit the window while keeping the aspect ratio.

[deanlee]

First Step: Migrate Onroad Widgets to Raylib 1.Organize the existing code by encapsulating related functions into specific classes to improve clarity and maintainability, The new class will expose only two public functions: updateState() and draw(), facilitating a smoother transition to Raylib.

• [x] https://github.com/commaai/openpilot/pull/33457
• [x] https://github.com/commaai/openpilot/pull/33459
• [x] https://github.com/commaai/openpilot/pull/33375
• [x] https://github.com/commaai/openpilot/pull/33458
• [x] https://github.com/commaai/openpilot/pull/33702
• [ ] https://github.com/commaai/openpilot/pull/33773

After step 1, the AnnotatedCameraWidget will function as a container for various components such as HudRenderer, AlertRenderer, DriverMonitorRenderer, ModelRenderer, and others. Its responsibilities will include updating transformations, managing window size, and handling user input. It will not handle state updates or rendering directly; instead, these tasks will be delegated to the drawing functions of its contained classes.

Following Step 1, we can also remove the UIScene struct from uiState to streamline it, allowing uiState to hold only the most essential state information required by the entire application.

2. Migrate to raylib

• [x] https://github.com/commaai/openpilot/pull/33738 (Introduced a new raylib/util.cc file containing utility functions to manage common tasks such as font handling, application initialization, and other reusable operations.)
• [ ] refactor selfdrive/ui/qt/text using raylib. (Implement Raylib-based ScrollArea in this step, or use the raygui lib)
• [ ] https://github.com/commaai/openpilot/pull/33756
• [ ] https://github.com/commaai/openpilot/pull/33808
• [ ] https://github.com/commaai/openpilot/pull/34360
• [x] Replace QtNetwork with libcurl (https://github.com/commaai/openpilot/pull/33851)
• [ ] refactor selfdrive/ui/qt/setup/updater using raylib. (implement Raylib-based Wifi manager)
• [ ] Port the rendering functions of the newly organized classes from Qt to raylib, ensuring better performance and seamless integration with raylib's rendering capabilities.

Tools:

• [x] https://github.com/commaai/openpilot/pull/33387

Misc.

• [x] https://github.com/commaai/openpilot/pull/33401
• [x] https://github.com/commaai/openpilot/pull/33395
• [x] https://github.com/commaai/openpilot/pull/33394
• [x] https://github.com/commaai/openpilot/pull/33409
• [x] https://github.com/commaai/openpilot/pull/33416
• [x] https://github.com/commaai/openpilot/pull/33408
• [x] https://github.com/commaai/openpilot/pull/33389

The final directory structure might look like this:

alerts.cc – Handles onroad alerts.
dmon.cc – driver monitor updates and rendering.
hud.cc – HUD updates and rendering.
model.cc – model updates and rendering.
annotated_camera.cc – container for update and rending onroad window.
onroad_home.cc – container for the Raylib window.
raylib_helpers.cc – helper functions for tasks such as managing fonts and drawing text.

Here's a rough draft, written from scratch (interface issues can be addressed later). Although there is a lot of temporary code and commented-out sections with no cleanup yet, approximately 600 lines of onroad code have already been reduced.

[deanlee]

In my opinion, the current UI structure and display elements are relatively simple, making migration straightforward. The main challenges involve replacing the qdbus dependency with libdbus for the WiFi manager and implementing multi-language support. We need to create a tool similar to Qt’s lupdate to extract translatable strings and generate test cases. Once these tasks are completed, the migration will be quick and can likely be done in one go.

[sunnyhaibin]

@mitchellgoffpc I see that you are working on porting UI to PyQt5 here: https://github.com/commaai/openpilot/tree/python-ui

Is this something you're experimenting with, or part of the roadmap?

[mitchellgoffpc]

Just something I'm experimenting with

[deanlee]

I’ve moved the discussion about the migration of wifimanager from this comment to this thread for easier discussion:

Would switching to sd-bus with systemd be feasible? sd-bus provides a higher-level API and better integration with systemd, which could potentially simplify development. Alternatively, could we consider using something simpler like wpa_supplicant, or is libdbus still the necessary base for our implementation?

[adeebshihadeh]

I’m open to it. Want to open a small example PR with it? I also found https://github.com/martinhaefner/simppl

Given how complicated dbus is, I might even be open to using nmcli…

[deanlee]

Here is a demo WifiManager: https://github.com/commaai/openpilot/pull/34364

[deanlee]

Raylib doesn’t display special characters like ⌫, →, or non-ASCII characters correctly by default because it lacks a font manager like Qt. Qt’s QTextEngine automatically loads fonts for various languages and symbols, including control characters and emojis.

In Raylib, fonts need to be manually loaded, and without the right ones, non-ASCII characters show up as "??". We need to implement a simple font management system to handle this, making the migration from Qt to Raylib smoother. Even without Raylib, removing Qt still means we must manage fonts manually.

Anyone who can contribute would be greatly helpful.

Resolving the character display issue in Raylib's keyboard (which corresponds to InputDialog::InputDialog in Qt) could be a good starting point. You can check out the implementation here: https://github.com/deanlee/openpilot/blob/raylib_software_keyboard/system/ui/raylib/controls/keyboard.cc

[adeebshihadeh]

@deanlee before we get too far in the Raylib rewrite, I think we need to check two more things:

• [x] can we make a good scrolling demo (ideally without GPU)? Qt's performance is one of its biggest problems, and this isn't worth doing if it doesn't address this
• [x] can we write the UI in python with raylib bindings? All python would be great, but even C++ widgets assembled with Python is fine too

[adeebshihadeh]

@maxime-desroches showed me a good scrolling demo, so we're good to go now.

@deanlee I suggest we proceed like this:

1. Move over spinner, updater, and the rest of the auxiliary GUIs. These need to stay in C++ (and ideally statically linked) to avoid issues when upgrading AGNOS.
2. Start porting over the rest of ui. I think we want to do this in Python, but this means we won't be able to do it incrementally. But I think that's fine.

[adeebshihadeh]

Actually maybe Python is fine for #1 too, and we just guarantee to always include a compatible Python raylib version in AGNOS.

[deanlee]

Actually maybe Python is fine for #1 too, and we just guarantee to always include a compatible Python raylib version in AGNOS.

Using Python for all GUIs makes perfect sense. We can implement a Python-based WiFi manager that seamlessly integrates with both the main UI and auxiliary GUIs.

[deanlee]

Move over spinner, updater, and the rest of the auxiliary GUIs:

• [x] spinner & text window https://github.com/commaai/openpilot/pull/34583
• [x] keyboard https://github.com/commaai/openpilot/pull/34584
• [x] wifi manager
• [ ] updater

[adeebshihadeh]

installer and updater do need to work with old versions of AGNOS, but we can save those for last.

[adeebshihadeh]

Also now that we've merged a few of the UIs, I think it's time to restate the goals.

Goals:

1. Maintainability: C++ -> Python, long build time -> no build time, big dependency -> minimal one for better portability. All huge wins.
2. UI off the GPU: https://github.com/commaai/openpilot/issues/30294
3. Polish: design aside, the current UI is janky with its poor scrolling, variable FPS, and a bunch of other little things. 1 will help a lot here too; more iterations means more polish.

---

@deanlee Given those goals, I think we should take the three UIs we've merged already and polish them up before continuing. This will set a good foundation for the rest of the rewrite:

• GuiApplication
  • [ ] make sure it's not using the GPU at all
  • [x] add monitoring for the FPS and log when it's less than 60
  • [x] add DEBUG_FPS=1: an environment variable for showing the FPS
  • [x] add STRICT_MODE=1: kills the UI if it drops too much below 60 FPS. maybe master runs in this mode all the time?
  • [ ] generic scaling across displays
• make the styles match the old UI
• make the scrolling in text feel like an iPhone. make an inertial scroller widget?

[adeebshihadeh]

@deanlee are we ready to move the small UIs (spinner, text, etc.) to the new versions? It'd be a great step to start shipping some raylib UIs to production!

[adeebshihadeh]

@deanlee the logging isn't filling in the format strings. I just noticed this since I'm fixing up the macOS support.

raylib: TEXTURE: [ID %i] Texture loaded successfully (%ix%i | %s | %i mipmaps)
raylib: FONT: Data loaded successfully (%i pixel size | %i glyphs)
raylib: FILEIO: [%s] File loaded successfully
raylib: TEXTURE: [ID %i] Texture loaded successfully (%ix%i | %s | %i mipmaps)
raylib: FONT: Data loaded successfully (%i pixel size | %i glyphs)
raylib: SHADER: [ID %i] Failed to compile vertex shader code
raylib: SHADER: [ID %i] Compile error: %s
raylib: SHADER: [ID %i] Failed to compile fragment shader code
raylib: SHADER: [ID %i] Compile error: %s

[deanlee]

the logging isn't filling in the format strings. I just noticed this since I'm fixing up the macOS support.

@adeebshihadeh : This issue is resolved in PR https://github.com/commaai/openpilot/pull/35561

[sshane]

Moving here https://github.com/commaai/openpilot/issues/36206