waiting for controls to start after update to 0.9.7

[jyoung8607]

Describe the bug

After this user received an automatic upgrade from 0.9.6-release to 0.9.7-release, they experienced "Waiting for controls to start" each time they tried to drive.

Tracebacks in the logs show core affinity couldn't be set for some critical processes, which implies one or more of the 845 performance cores might not have come online when transitioning offroad->onroad.

Ignition/onroad cycles did not clear the problem, but a power cycle did clear the problem.

Provide a route where the issue occurs

b08418a961021457/00000001--8dec252c1f

openpilot version

0.9.7

Additional info

[0.005306] system/loggerd/loggerd.cc:218 - loggerd_thread: logging accelerometer
[0.006482] process.py:291 - : starting python selfdrive.monitoring.dmonitoringd
[0.006877] process.py:46 - : crash
Traceback (most recent call last):
  File "/data/openpilot/openpilot/system/manager/process.py", line 40, in launcher
    mod.main()
  File "/data/openpilot/selfdrive/controls/controlsd.py", line 844, in main
    config_realtime_process(4, Priority.CTRL_HIGH)
  File "/data/openpilot/openpilot/common/realtime.py", line 44, in config_realtime_process
    set_core_affinity(c)
  File "/data/openpilot/openpilot/common/realtime.py", line 37, in set_core_affinity
    os.sched_setaffinity(0, cores)
OSError: [Errno 22] Invalid argument
[0.006926] process.py:46 - : crash
Traceback (most recent call last):
  File "/data/openpilot/openpilot/system/manager/process.py", line 40, in launcher
    mod.main()
  File "/data/openpilot/selfdrive/controls/controlsd.py", line 844, in main
    config_realtime_process(4, Priority.CTRL_HIGH)
  File "/data/openpilot/openpilot/common/realtime.py", line 44, in config_realtime_process
    set_core_affinity(c)
  File "/data/openpilot/openpilot/common/realtime.py", line 37, in set_core_affinity
    os.sched_setaffinity(0, cores)
OSError: [Errno 22] Invalid argument
[0.006993] system/loggerd/loggerd.cc:218 - loggerd_thread: logging accelerometer2
[0.007046] system/loggerd/loggerd.cc:218 - loggerd_thread: logging androidLog
[0.007093] system/loggerd/loggerd.cc:218 - loggerd_thread: logging cameraOdometry

[jackking444]

I get the same issue Traceback (most recent call last): File "/data/openpilot/selfdrive/modeld/modeld.py", line 419, in main(demo=args.demo) File "/data/openpilot/selfdrive/modeld/modeld.py", line 174, in main config_realtime_process(7, 54) File "/data/openpilot/openpilot/common/realtime.py", line 46, in config_realtime_process set_core_affinity(c) File "/data/openpilot/openpilot/common/realtime.py", line 39, in set_core_affinity os.sched_setaffinity(0, cores) OSError: [Errno 22] Invalid argument

[jollycar]

I have the same problem. If i start the car, I get the comma.ai logo, then the dash-cam turns on, then I get "Waiting for controls to start" for 2-3 seconds, then it displays the menu. It does not go back to the dash-cam.

Extra info: I upgraded from 0.9.6 to 0.9.7 the first day it came out. The first time the problem occurred was after a few months driving problem free with 0.9.7. I never installed any un-official software, only openpilot.

The start of all problems was one day, the screen went totally black (no way to get a picture whatever I tried) and I had to let the battery drain.

My car is Toyota Corolla Hybrid 2021 (Europe)

I have 2 workarounds

1.Let the battery drain (by removing the usb-c cable) and then connect and drive. 2. Start driving with the usb-c connected, then after 5 minutes of driving, remove the usb-c cable, then after another 5 minutes reconnect

I did a reset of the device by tapping the screen.

If I manage to get logging, I will add it here

[jollycar]

I have no clue how to get logging of my device. I have looked in the comma.ai site in the FAQ and WIKI, I have looked in Discord but I have tried my best to find info about getting logs off the device. No pinned messages, did not find channels for debugging/etc.

[adeebshihadeh]

rlogs are gone for that route, and there's nothing good in the bootlog. Gonna close until it pops up again. May have been a one time 0.9.6 -> 0.9.7 warm boot thing.

[tonesto7]

@adeebshihadeh I've been seeing this error on recent builds of Sunnypilot:

[OSError: [Errno 22] Invalid argument
File "openpilot/system/manager/process.py", line 41, in launcher
mod.main()
File "selfdrive/modeld/dmonitoringmodeld.py", line 137, in main
config_realtime_process(7, 5)
File "openpilot/common/realtime.py", line 40, in config_realtime_process
set_core_affinity(c)
File "openpilot/common/realtime.py", line 32, in set_core_affinity
os.sched_setaffinity(0, cores)
https://github.com/search?q=repo%3Acommaai%2Fopenpilot%20config_realtime_process&type=code

Perhaps I don't fully understand how the affinity settings work, but I know the first argument of config_realtime_process is the number of cores.

The dmonitoringmodeld.py is hard-coded to 7 cores. The comma3x only has 4 cores.

[jyoung8607]

The dmonitoringmodeld.py is hard-coded to 7 cores. The comma3x only has 4 cores.

It has eight. If it's offroad you'll only see four in top/htop because the perf cores get offlined for power savings.

Edit: sorry I broke your earlier comment, hit the wrong button to reply.

[tonesto7]

@jyoung8607 Thanks for the reply! This points me in a direction to debug.