test_models: run on route list

[sshane]

Includes one segment from every dongle id.

Catching some bugs:

FAILED run_test_models.py::TestCarModel_11_NISSAN_LEAF_2018::test_panda_safety_carstate - AssertionError: 1 is not false : panda safety doesn't agree with openpilot: {'standstill': 14}
FAILED run_test_models.py::TestCarModel_26_HYUNDAI_SONATA_2020::test_panda_safety_carstate - AssertionError: 1 is not false : panda safety doesn't agree with openpilot: {'cruiseState': 2}
FAILED run_test_models.py::TestCarModel_100_TOYOTA_HIGHLANDER_2017::test_panda_safety_carstate - AssertionError: 1 is not false : panda safety doesn't agree with openpilot: {'cruiseState': 1}
FAILED run_test_models.py::TestCarModel_174_KIA_EV6_2022::test_car_interface - AssertionError: 5798 != 0
FAILED run_test_models.py::TestCarModel_201_TOYOTA_PRIUS_2017::test_panda_safety_carstate - AssertionError: 1 is not false : panda safety doesn't agree with openpilot: {'cruiseState': 3}
FAILED run_test_models.py::TestCarModel_283_NISSAN_ROGUE_2019::test_panda_safety_rx_valid - AssertionError: False is not true
FAILED run_test_models.py::TestCarModel_283_NISSAN_ROGUE_2019::test_car_interface - AssertionError: 348 != 0
FAILED run_test_models.py::TestCarModel_321_HONDA_CIVIC_2016::test_panda_safety_carstate - AssertionError: 1 is not false : panda safety doesn't agree with openpilot: {'controlsAllowed': 1}
FAILED run_test_models.py::TestCarModel_620_TOYOTA_PRIUS_2017::test_car_interface - AssertionError: 1881 != 0
FAILED run_test_models.py::TestCarModel_620_TOYOTA_PRIUS_2017::test_panda_safety_rx_valid - AssertionError: False is not true
FAILED run_test_models.py::TestCarModel_633_HONDA_ODYSSEY_2018::test_car_interface - AssertionError: 5799 != 0
FAILED run_test_models.py::TestCarModel_633_HONDA_ODYSSEY_2018::test_panda_safety_carstate - AssertionError: 2 is not false : panda safety doesn't agree with openpilot: {'brakePressed': 1, 'mainOn': 544}
FAILED run_test_models.py::TestCarModel_633_HONDA_ODYSSEY_2018::test_panda_safety_rx_valid - AssertionError: False is not true
FAILED run_test_models.py::TestCarModel_350_KIA_EV6_2022::test_car_interface - AssertionError: 5800 != 0
FAILED run_test_models.py::TestCarModel_698_NISSAN_ALTIMA_2020::test_panda_safety_carstate - AssertionError: 1 is not false : panda safety doesn't agree with openpilot: {'standstill': 10}
FAILED run_test_models.py::TestCarModel_575_TOYOTA_PRIUS_2017::test_car_interface - AssertionError: 19 != 0
FAILED run_test_models.py::TestCarModel_413_HYUNDAI_IONIQ_5_2022::test_car_interface - AssertionError: 5799 != 0
FAILED run_test_models.py::TestCarModel_742_TOYOTA_COROLLA_2017::test_car_interface - AssertionError: 17 != 0
FAILED run_test_models.py::TestCarModel_553_HONDA_PILOT_2017::test_panda_safety_carstate - AssertionError: 1 is not false : panda safety doesn't agree with openpilot: {'controlsAllowed': 1}
FAILED run_test_models.py::TestCarModel_755_KIA_EV6_2022::test_car_interface - AssertionError: 5798 != 0
FAILED run_test_models.py::TestCarModel_755_KIA_EV6_2022::test_panda_safety_rx_valid - AssertionError: False is not true
FAILED run_test_models.py::TestCarModel_432_HYUNDAI_IONIQ_5_2022::test_car_interface - AssertionError: 5799 != 0
FAILED run_test_models.py::TestCarModel_906_LEXUS_RX_2016::test_panda_safety_carstate - AssertionError: 1 is not false : panda safety doesn't agree with openpilot: {'cruiseState': 1}
FAILED run_test_models.py::TestCarModel_784_HYUNDAI_KONA_ELECTRIC_2019::test_car_interface - AssertionError: 5799 != 0
FAILED run_test_models.py::TestCarModel_784_HYUNDAI_KONA_ELECTRIC_2019::test_panda_safety_rx_valid - AssertionError: False is not true

[sshane]

How to run:

INTERNAL_SEG_LIST="selfdrive/car/tests/test_models_segs.txt" pytest -n32 --dist=loadscope ./test_models.py

Total dongles/segments: 1948

On the jenkins machine, it takes approx. 150 mins with 5 jobs (tried matrix approach, but you can't put matrix inside parallel)

Or approx. 20 minutes with pytest and 42 jobs (copied from the number of scons jobs)

[sshane]

What it looks like in jenkins: https://jenkins.comma.life/blue/organizations/jenkins/openpilot/detail/big-test-models-temp/3/pipeline