openpilot
openpilot copied to clipboard
Published
20 hours ago •
commaai
commaai
raylib spinner slow while compiling
trafficstars
it will freeze for seconds at a time while compiling, qt never lagged this bad. @incognitojam can you check if there's anything obviously wrong with the rl spinner?
Also what is its CPU usage vs the qt one?
Using pidstat it looks like there is a huge spike in CPU usage when starting the Raylib spinner, and it "idles" at 2-3% on PC. In comparison the Qt spinner load is much lower.
On device Raylib is even worse, it takes 3 seconds to initialise raylib and load the assets, and the "idle" usage is 12%.
Old results
Raylib (device)
$ pidstat -ruh 1 10 -e python system/ui/spinner.py
# Time UID PID %usr %system %guest %wait %CPU CPU minflt/s majflt/s VSZ RSS %MEM Command
11:54:30 AM 1000 54389 93.07 36.63 0.00 0.00 129.70 2 7894.06 0.00 216452 28340 0.77 python
11:54:31 AM 1000 54389 91.00 7.00 0.00 0.00 98.00 2 6114.00 0.00 331204 51576 1.40 python
11:54:32 AM 1000 54389 93.00 6.00 0.00 0.00 99.00 2 8411.00 0.00 348652 78732 2.13 python
11:54:33 AM 1000 54389 40.00 2.00 0.00 0.00 42.00 0 242.00 0.00 444380 79772 2.16 python
11:54:34 AM 1000 54389 10.00 2.00 0.00 0.00 12.00 0 19.00 0.00 444380 79772 2.16 python
11:54:35 AM 1000 54389 13.00 0.00 0.00 0.00 13.00 0 18.00 0.00 444412 79772 2.16 python
11:54:36 AM 1000 54389 12.00 0.00 0.00 0.00 12.00 0 18.00 0.00 444412 80020 2.17 python
11:54:37 AM 1000 54389 11.00 1.00 0.00 0.00 12.00 0 18.00 0.00 444412 80020 2.17 python
Qt (device)
$ pidstat -ruh 1 5 -e python common/spinner.py
# Time UID PID %usr %system %guest %wait %CPU CPU minflt/s majflt/s VSZ RSS %MEM Command
11:58:37 AM 1000 54462 9.90 3.96 0.00 0.00 13.86 0 2360.40 0.00 19052 8760 0.24 python
11:58:38 AM 1000 54462 0.00 0.00 0.00 0.00 0.00 0 0.00 0.00 19052 8760 0.24 python
11:58:39 AM 1000 54462 0.00 0.00 0.00 0.00 0.00 0 0.00 0.00 19052 8760 0.24 python
11:58:40 AM 1000 54462 0.00 0.00 0.00 0.00 0.00 0 0.00 0.00 19052 8760 0.24 python
11:58:41 AM 1000 54462 0.00 0.00 0.00 0.00 0.00 0 0.00 0.00 19052 8760 0.24 python
Raylib (PC)
# Time UID PID %usr %system %guest %wait %CPU CPU minflt/s majflt/s VSZ RSS %MEM Command
12:47:17 1000 50823 125.00 5.00 0.00 0.00 130.00 4 20293.00 0.00 1952876 231324 0.35 python
12:47:18 1000 50823 4.00 1.00 0.00 0.00 5.00 1 3.00 0.00 1952876 231324 0.35 python
12:47:19 1000 50823 4.00 1.00 0.00 0.00 5.00 1 2.00 0.00 1952876 231324 0.35 python
12:47:20 1000 50823 5.00 0.00 0.00 0.00 5.00 1 3.00 0.00 1952876 231324 0.35 python
12:47:21 1000 50823 5.00 1.00 0.00 0.00 6.00 1 3.00 0.00 1952876 231324 0.35 python
Qt (PC)
# Time UID PID %usr %system %guest %wait %CPU CPU minflt/s majflt/s VSZ RSS %MEM Command
12:48:12 1000 51361 1.00 0.00 0.00 0.00 1.00 12 1789.00 0.00 38652 18360 0.03 python
12:48:13 1000 51361 0.00 0.00 0.00 0.00 0.00 12 0.00 0.00 38652 18360 0.03 python
12:48:14 1000 51361 0.00 0.00 0.00 0.00 0.00 12 0.00 0.00 38652 18360 0.03 python
12:48:15 1000 51361 0.00 0.00 0.00 0.00 0.00 12 0.00 0.00 38652 18360 0.03 python
12:48:16 1000 51361 0.00 0.00 0.00 0.00 0.00 12 0.00 0.00 38652 18360 0.03 python
This is a big help for investigating - https://plugins.jetbrains.com/plugin/16536-line-profiler
It seems every instantiation of an rl.Vector2 or rl.Rectangle is slow - we can move these out of the render loop
I made a mistake by measuring common/spinner.py for Qt instead of the actual _spinner binary - its CPU is around 10-12% which makes Raylib seem less bad. I'm also making sure to run ./scripts/disable-powersave.py because otherwise the % are much higher.
| Script | Average CPU after init |
|---|---|
selfdrive/ui/_spinner (Qt) |
11% |
system/ui/spinner.py (with SetTargetFPS(30)) |
8-9% |
system/ui/spinner.py (no SetTargetFPS) |
3-4% |
| Build | htop reported CPU usage |
|---|---|
with raylib spinner (SetTargetFPS(30)) |
4.8-5.5% |
| with qt spinner | 13.2-15.2% |
Full results from on device testing
$ pidstat -ruh 1 10 -e ./selfdrive/ui/_spinner
Linux 4.9.103 (comma-6c65f4e7) 05/04/2025 _aarch64_ (8 CPU)
Using Wayland-EGL
qt.qpa.wayland: "wl-shell" is a deprecated shell extension, prefer using "xdg-shell-v6" or "xdg-shell" if supported by the compositor by setting the environment variable QT_WAYLAND_SHELL_INTEGRATION
Using the 'wl-shell' shell integration
# Time UID PID %usr %system %guest %wait %CPU CPU minflt/s majflt/s VSZ RSS %MEM Command
09:30:39 PM 1000 82739 22.00 23.00 0.00 0.00 45.00 0 26589.00 0.00 361120 102348 2.77 _spinner
09:30:40 PM 1000 82739 11.00 0.00 0.00 1.00 11.00 0 1.00 0.00 361120 102348 2.77 _spinner
09:30:41 PM 1000 82739 10.89 0.00 0.00 0.00 10.89 1 0.00 0.00 361120 102348 2.77 _spinner
09:30:42 PM 1000 82739 12.00 0.00 0.00 2.00 12.00 0 0.00 0.00 361120 102348 2.77 _spinner
09:30:43 PM 1000 82739 8.00 3.00 0.00 1.00 11.00 0 0.00 0.00 361120 102348 2.77 _spinner
09:30:44 PM 1000 82739 8.00 3.00 0.00 1.00 11.00 0 0.00 0.00 361120 102348 2.77 _spinner
$ pidstat -ruh 1 10 -e python system/ui/spinner.py # with SetTargetFPS(30)
Linux 4.9.103 (comma-6c65f4e7) 05/04/2025 _aarch64_ (8 CPU)
RAYLIB STATIC 5.5.0.2 LOADED
# Time UID PID %usr %system %guest %wait %CPU CPU minflt/s majflt/s VSZ RSS %MEM Command
09:39:49 PM 1000 82896 72.00 64.00 0.00 4.00 136.00 4 8860.00 11.00 384044 31800 0.86 python
INFO: Initializing raylib 5.6-dev
INFO: Platform backend: COMMA
INFO: Supported raylib modules:
INFO: > rcore:..... loaded (mandatory)
INFO: > rlgl:...... loaded (mandatory)
INFO: > rshapes:... loaded (optional)
INFO: > rtextures:. loaded (optional)
INFO: > rtext:..... loaded (optional)
INFO: > rmodels:... loaded (optional)
INFO: > raudio:.... loaded (optional)
gbm_create_device(156): Info: backend name is: msm_drm
INFO: COMMA: Using EGL version 1.5
INFO: COMMA: Found 1 valid EGL display configs
INFO: GL: Supported extensions count: 91
INFO: GL: OpenGL device information:
INFO: > Vendor: Qualcomm
INFO: > Renderer: Adreno (TM) 630
INFO: > Version: OpenGL ES 3.2
INFO: > GLSL: OpenGL ES GLSL ES 3.20
INFO: GL: VAO extension detected, VAO functions loaded successfully
INFO: GL: NPOT textures extension detected, full NPOT textures supported
INFO: GL: ETC1 compressed textures supported
INFO: GL: ASTC compressed textures supported
INFO: COMMA: Initialized successfully
INFO: TEXTURE: [ID 1] Texture loaded successfully (1x1 | R8G8B8A8 | 1 mipmaps)
INFO: TEXTURE: [ID 1] Default texture loaded successfully
INFO: SHADER: [ID 1] Vertex shader compiled successfully
INFO: SHADER: [ID 2] Fragment shader compiled successfully
INFO: SHADER: [ID 3] Program shader loaded successfully
INFO: SHADER: [ID 3] Default shader loaded successfully
INFO: RLGL: Render batch vertex buffers loaded successfully in RAM (CPU)
INFO: RLGL: Render batch vertex buffers loaded successfully in VRAM (GPU)
INFO: RLGL: Default OpenGL state initialized successfully
INFO: TEXTURE: [ID 2] Texture loaded successfully (128x128 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Default font loaded successfully (224 glyphs)
INFO: SYSTEM: Working Directory: /data/openpilot
INFO: TIMER: Target time per frame: 33.333 milliseconds
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-Black.ttf] File loaded successfully
INFO: TEXTURE: [ID 3] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-Bold.ttf] File loaded successfully
INFO: TEXTURE: [ID 4] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-ExtraBold.ttf] File loaded successfully
INFO: TEXTURE: [ID 5] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-ExtraLight.ttf] File loaded successfully
INFO: TEXTURE: [ID 6] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-Medium.ttf] File loaded successfully
INFO: TEXTURE: [ID 7] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-Regular.ttf] File loaded successfully
INFO: TEXTURE: [ID 8] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-SemiBold.ttf] File loaded successfully
INFO: TEXTURE: [ID 9] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-Thin.ttf] File loaded successfully
INFO: TEXTURE: [ID 10] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/img_spinner_comma.png] File loaded successfully
INFO: IMAGE: Data loaded successfully (848x848 | R8G8B8A8 | 1 mipmaps)
INFO: TEXTURE: [ID 11] Texture loaded successfully (360x360 | R8G8B8A8 | 1 mipmaps)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/img_spinner_track.png] File loaded successfully
09:39:50 PM 1000 82896 83.00 8.00 0.00 0.00 91.00 0 12981.00 1.00 512644 76216 2.06 python
INFO: IMAGE: Data loaded successfully (848x848 | R8G8B8A8 | 1 mipmaps)
INFO: TEXTURE: [ID 12] Texture loaded successfully (360x360 | R8G8B8A8 | 1 mipmaps)
09:39:51 PM 1000 82896 29.00 3.00 0.00 0.00 32.00 0 944.00 0.00 608372 79896 2.16 python
09:39:52 PM 1000 82896 6.00 2.00 0.00 0.00 8.00 0 18.00 0.00 608372 79896 2.16 python
09:39:53 PM 1000 82896 6.00 3.00 0.00 0.00 9.00 0 18.00 0.00 608404 79896 2.16 python
09:39:54 PM 1000 82896 6.00 2.00 0.00 0.00 8.00 0 19.00 0.00 608404 80156 2.17 python
09:39:55 PM 1000 82896 5.00 4.00 0.00 0.00 9.00 0 17.00 0.00 608404 80156 2.17 python
$ pidstat -ruh 1 10 -e python system/ui/spinner.py # without SetTargetFPS
Linux 4.9.103 (comma-6c65f4e7) 05/04/2025 _aarch64_ (8 CPU)
RAYLIB STATIC 5.5.0.2 LOADED
# Time UID PID %usr %system %guest %wait %CPU CPU minflt/s majflt/s VSZ RSS %MEM Command
09:42:05 PM 1000 82939 105.00 51.00 0.00 0.00 156.00 4 11433.00 0.00 398940 40920 1.11 python
INFO: Initializing raylib 5.6-dev
INFO: Platform backend: COMMA
INFO: Supported raylib modules:
INFO: > rcore:..... loaded (mandatory)
INFO: > rlgl:...... loaded (mandatory)
INFO: > rshapes:... loaded (optional)
INFO: > rtextures:. loaded (optional)
INFO: > rtext:..... loaded (optional)
INFO: > rmodels:... loaded (optional)
INFO: > raudio:.... loaded (optional)
gbm_create_device(156): Info: backend name is: msm_drm
INFO: COMMA: Using EGL version 1.5
INFO: COMMA: Found 1 valid EGL display configs
INFO: GL: Supported extensions count: 91
INFO: GL: OpenGL device information:
INFO: > Vendor: Qualcomm
INFO: > Renderer: Adreno (TM) 630
INFO: > Version: OpenGL ES 3.2
INFO: > GLSL: OpenGL ES GLSL ES 3.20
INFO: GL: VAO extension detected, VAO functions loaded successfully
INFO: GL: NPOT textures extension detected, full NPOT textures supported
INFO: GL: ETC1 compressed textures supported
INFO: GL: ASTC compressed textures supported
INFO: COMMA: Initialized successfully
INFO: TEXTURE: [ID 1] Texture loaded successfully (1x1 | R8G8B8A8 | 1 mipmaps)
INFO: TEXTURE: [ID 1] Default texture loaded successfully
INFO: SHADER: [ID 1] Vertex shader compiled successfully
INFO: SHADER: [ID 2] Fragment shader compiled successfully
INFO: SHADER: [ID 3] Program shader loaded successfully
INFO: SHADER: [ID 3] Default shader loaded successfully
INFO: RLGL: Render batch vertex buffers loaded successfully in RAM (CPU)
INFO: RLGL: Render batch vertex buffers loaded successfully in VRAM (GPU)
INFO: RLGL: Default OpenGL state initialized successfully
INFO: TEXTURE: [ID 2] Texture loaded successfully (128x128 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Default font loaded successfully (224 glyphs)
INFO: SYSTEM: Working Directory: /data/openpilot
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-Black.ttf] File loaded successfully
INFO: TEXTURE: [ID 3] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-Bold.ttf] File loaded successfully
INFO: TEXTURE: [ID 4] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-ExtraBold.ttf] File loaded successfully
INFO: TEXTURE: [ID 5] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-ExtraLight.ttf] File loaded successfully
INFO: TEXTURE: [ID 6] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-Medium.ttf] File loaded successfully
INFO: TEXTURE: [ID 7] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-Regular.ttf] File loaded successfully
INFO: TEXTURE: [ID 8] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-SemiBold.ttf] File loaded successfully
INFO: TEXTURE: [ID 9] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/fonts/Inter-Thin.ttf] File loaded successfully
INFO: TEXTURE: [ID 10] Texture loaded successfully (1024x1024 | GRAY_ALPHA | 1 mipmaps)
INFO: FONT: Data loaded successfully (120 pixel size | 95 glyphs)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/img_spinner_comma.png] File loaded successfully
INFO: IMAGE: Data loaded successfully (848x848 | R8G8B8A8 | 1 mipmaps)
INFO: TEXTURE: [ID 11] Texture loaded successfully (360x360 | R8G8B8A8 | 1 mipmaps)
INFO: FILEIO: [/data/openpilot/selfdrive/assets/img_spinner_track.png] File loaded successfully
INFO: IMAGE: Data loaded successfully (848x848 | R8G8B8A8 | 1 mipmaps)
INFO: TEXTURE: [ID 12] Texture loaded successfully (360x360 | R8G8B8A8 | 1 mipmaps)
# Time UID PID %usr %system %guest %wait %CPU CPU minflt/s majflt/s VSZ RSS %MEM Command
09:42:06 PM 1000 82939 78.00 12.00 0.00 0.00 90.00 0 11365.00 0.00 608388 79928 2.16 python
09:42:07 PM 1000 82939 3.00 1.00 0.00 0.00 4.00 0 19.00 0.00 608388 79928 2.16 python
09:42:08 PM 1000 82939 3.00 0.00 0.00 0.00 3.00 0 18.00 0.00 608388 79928 2.16 python
09:42:09 PM 1000 82939 3.00 1.00 0.00 0.00 4.00 0 18.00 0.00 608420 80188 2.17 python
09:42:10 PM 1000 82939 2.00 2.00 0.00 0.00 4.00 0 18.00 0.00 608420 80188 2.17 python
I have seen the spinner lag before, but I can't repro it by starting processes to use CPU - even with 100% cpu activity the spinner is managing 30fps. Could it be GPU related when doing the model compile with tinygrad?
From a clean scons it completely froze
https://github.com/user-attachments/assets/101fc67f-aeca-4b1a-aea3-749d91f5c721
When I unplugged it after maybe 10 mins, it resumed near the end so it likely was still compiling?
I’m unable to reproduce this issue on my device. @incognitojam : Could this be related to a timing issue on device? The latest code uses rl.get_frame_time() to adjust the rotation angle.
clang++ -o rednose/helpers/ekf_sym_pyx.so -pthread -shared -Wl,-rpath=/data/openpilot/third_party/acados/larch64/lib -Wl,-rpath=/usr/local/lib rednose/helpers/ekf_sym_pyx.o rednose/helpers/ekf_load.os rednose/helpers/ekf_sym.os -L/usr/local/lib -L/system/vendor/lib64 -Lthird_party/acados/larch64/lib -Lthird_party/libyuv/larch64/lib -L/usr/lib/aarch64-linux-gnu -Lmsgq_repo -Lthird_party -Lselfdrive/pandad -Lcommon -Lrednose/helpers -ldl common/libcommon.a -ljson11 -lzmq -lzmq
clang -o selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_cost/long_cost_y_e_hess.os -c -std=gnu11 -DQCOM2 -mcpu=cortex-a57 -DACADOS_WITH_QPOASES -g -fPIC -O2 -Wunused -Werror -Wshadow -Wno-unknown-warning-option -Wno-inconsistent-missing-override -Wno-c99-designator -Wno-reorder-init-list -Wno-vla-cxx-extension -DQCOM2 -mcpu=cortex-a57 -Wno-unused -fPIC -Ithird_party/opencl/include -I. -Ithird_party/acados/include -Ithird_party/acados/include/blasfeo/include -Ithird_party/acados/include/hpipm/include -Ithird_party/catch2/include -Ithird_party/libyuv/include -Ithird_party/json11 -Ithird_party/linux/include -Ithird_party -Imsgq selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_cost/long_cost_y_e_hess.c
FPS dropped below 60: 47
clang -o selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/libacados_ocp_solver_long.so -Wl,--as-needed -Wl,--no-undefined -Wl,--disable-new-dtags -shared -Wl,-rpath=/data/openpilot/third_party/acados/larch64/lib -Wl,-rpath=/usr/local/lib selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/acados_solver_long.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_model/long_expl_ode_fun.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_model/long_expl_vde_forw.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_cost/long_cost_y_fun.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_cost/long_cost_y_fun_jac_ut_xt.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_cost/long_cost_y_hess.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_cost/long_cost_y_e_fun.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_cost/long_cost_y_e_fun_jac_ut_xt.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_cost/long_cost_y_e_hess.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_cost/long_cost_y_0_fun.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_cost/long_cost_y_0_fun_jac_ut_xt.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_cost/long_cost_y_0_hess.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_constraints/long_constr_h_fun.os selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/long_constraints/long_constr_h_fun_jac_uxt_zt.os -L/usr/local/lib -L/system/vendor/lib64 -Lthird_party/acados/larch64/lib -Lthird_party/libyuv/larch64/lib -L/usr/lib/aarch64-linux-gnu -Lmsgq_repo -Lthird_party -Lselfdrive/pandad -Lcommon -Lrednose/helpers -lm -lacados -lhpipm -lblasfeo -lqpOASES_e
python3 /data/openpilot/selfdrive/modeld/get_model_metadata.py /data/openpilot/selfdrive/modeld/models/driving_vision.onnx
PYTHONPATH=":/data/openpilot/tinygrad_repo" QCOM=1 python3 /data/openpilot/tinygrad_repo/examples/openpilot/compile3.py /data/openpilot/selfdrive/modeld/models/driving_vision.onnx /data/openpilot/selfdrive/modeld/models/driving_vision_tinygrad.pkl
clang -o selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/acados_ocp_solver_pyx.o -c -std=gnu11 -DQCOM2 -mcpu=cortex-a57 -g -fPIC -O2 -Wunused -Wshadow -Wno-unknown-warning-option -Wno-inconsistent-missing-override -Wno-c99-designator -Wno-reorder-init-list -Wno-vla-cxx-extension -DQCOM2 -mcpu=cortex-a57 -Wno-#warnings -Wno-shadow -Wno-deprecated-declarations -Ithird_party/opencl/include -I. -Ithird_party/acados/include -Ithird_party/acados/include/blasfeo/include -Ithird_party/acados/include/hpipm/include -Ithird_party/catch2/include -Ithird_party/libyuv/include -Ithird_party/json11 -Ithird_party/linux/include -Ithird_party -Imsgq -I/usr/include/python3.12 -I/usr/local/venv/lib/python3.12/site-packages/numpy/_core/include selfdrive/controls/lib/longitudinal_mpc_lib/c_generated_code/acados_ocp_solver_pyx.c
saved metadata to /data/openpilot/selfdrive/modeld/models/driving_vision_metadata.pkl
clang++ -o selfdrive/pandad/panda.o -c -std=c++1z -DQCOM2 -mcpu=cortex-a57 -g -fPIC -O2 -Wunused -Werror -Wshadow -Wno-unknown-warning-option -Wno-inconsistent-missing-override -Wno-c99-designator -Wno-reorder-init-list -Wno-vla-cxx-extension -DQCOM2 -mcpu=cortex-a57 -Ithird_party/opencl/include -I. -Ithird_party/acados/include -Ithird_party/acados/include/blasfeo/include -Ithird_party/acados/include/hpipm/include -Ithird_party/catch2/include -Ithird_party/libyuv/include -Ithird_party/json11 -Ithird_party/linux/include -Ithird_party -Imsgq selfdrive/pandad/panda.cc
loaded model
FPS dropped below 60: 45
created tensors
run 0
FPS dropped below 60: 47
clang -o selfdrive/controls/lib/lateral_mpc_lib/c_generated_code/acados_ocp_solver_pyx.so -pthread -shared /data/openpilot/selfdrive/controls/lib/lateral_mpc_lib/c_generated_code/libacados_ocp_solver_lat.so -Wl,-rpath=/data/openpilot/third_party/acados/larch64/lib -Wl,-rpath=/usr/local/lib selfdrive/controls/lib/lateral_mpc_lib/c_generated_code/acados_ocp_solver_pyx.o -L/usr/local/lib -L/system/vendor/lib64 -Lthird_party/acados/larch64/lib -Lthird_party/libyuv/larch64/lib -L/usr/lib/aarch64-linux-gnu -Lmsgq_repo -Lthird_party -Lselfdrive/pandad -Lcommon -Lrednose/helpers
clang++ -o selfdrive/pandad/panda_comms.o -c -std=c++1z -DQCOM2 -mcpu=cortex-a57 -g -fPIC -O2 -Wunused -Werror -Wshadow -Wno-unknown-warning-option -Wno-inconsistent-missing-override -Wno-c99-designator -Wno-reorder-init-list -Wno-vla-cxx-extension -DQCOM2 -mcpu=cortex-a57 -Ithird_party/opencl/include -I. -Ithird_party/acados/include -Ithird_party/acados/include/blasfeo/include -Ithird_party/acados/include/hpipm/include -Ithird_party/catch2/include -Ithird_party/libyuv/include -Ithird_party/json11 -Ithird_party/linux/include -Ithird_party -Imsgq selfdrive/pandad/panda_comms.cc
scheduled 405 kernels in 2896.76 ms
FPS dropped below 60: 46
FPS dropped below 60: 9
FPS dropped below 60: 4
FPS dropped below 60: 2
FPS dropped below 60: 1
FPS dropped below 60: 1
FPS dropped below 60: 1
FPS dropped below 60: 1
FPS dropped below 60: 1
FPS dropped below 60: 0
FPS dropped below 60: 0
FPS dropped below 60: 0
FPS dropped below 60: 0
FPS dropped below 60: 0
FPS dropped below 60: 0
FPS dropped below 60: 0
scons: *** [selfdrive/locationd/models/generated/car.os] Error -9
FPS dropped below 60: 52