Speedup `test_car_interfaces`

[adeebshihadeh]

Takes ~1-2s per car. That's a big hit since we support nearly 300 different cars.

Goal is <=0.2s avg and <1s max on the CI machine, with the same test coverage. There's likely two or three things slowing this test down that'll be obvious with a bit of profiling.

Command to run:

pytest --durations=0 selfdrive/car/tests/test_car_interfaces.py 

Current output on my workstation

batman:sixpilot$ pytest --durations=10 selfdrive/car/tests/test_car_interfaces.py
=============================================================== test session starts ===============================================================
platform linux -- Python 3.11.4, pytest-8.2.0, pluggy-1.5.0
Using --randomly-seed=431984017
rootdir: /dev/shm/openpilot/sixpilot
configfile: pyproject.toml
plugins: randomly-3.15.0, subtests-0.12.1, anyio-4.3.0, cov-5.0.0, xdist-3.6.1, cpp-2.5.0, hypothesis-6.47.5, mock-3.14.0, timeout-2.3.1, flaky-3.8.1, asyncio-0.23.6
asyncio: mode=Mode.STRICT
32 workers [209 items]
........................................................................................................................................... [ 65%]
......................................................................                                                                      [100%]
============================================================== slowest 10 durations ===============================================================
2.17s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_027_FORD_FOCUS_MK4
2.17s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_138_MAZDA_CX9_2021
2.15s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_037_GENESIS_GV60_EV_1ST_GEN
2.05s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_147_SKODA_FABIA_MK4
1.97s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_094_HYUNDAI_VELOSTER
1.97s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_005_AUDI_Q3_MK2
1.87s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_059_HONDA_PILOT
1.83s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_113_KIA_OPTIMA_H_G4_FL
1.83s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_076_HYUNDAI_IONIQ_PHEV_2019
1.78s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_095_JEEP_GRAND_CHEROKEE
============================================================== 209 passed in 13.32s ===============================================================

[sshane]

A good place to optimize is the hypothesis example generation. Speeds are ~2x faster than our current version in version 4.51.0 and get slower somewhere between then and 4.55.3. The latest version (6.102.6) is even slower compared to our current version (6.47). You can root cause and open a PR to hypothesis or openpilot depending on where the optimization lies.

[deanlee]

Another option is to optimize get_fuzzy_car_interface_args by caching params_strategy to avoid regenerating strategies each time. Additionally, drawing st.dictionaries is quite slow; according to my tests, it can take up to 0.3 seconds.

[lsrugo]

Another option is to optimize get_fuzzy_car_interface_args by caching params_strategy to avoid regenerating strategies each time.

I tried this and it didn't seem to work.

Additionally, drawing st.dictionaries is quite slow; according to my tests, it can take up to 0.3 seconds.

Right so it's likely a problem in the underlying library

[geeth345]

Is this bounty still available? I'm looking to work on it but am confused if it has already been claimed by one of the PRs above

[Pious1020]

@adeebshihadeh, is this still open?

[c-gamble]

I'm interested in working on this but it's looking like this bounty has been (partially) solved? Is the bounty still open to go from~0.7s to <= 0.2s?

[sshane]

Getting this result on a AMD Ryzen Threadripper PRO 3955WX 16-Cores:

Using --randomly-seed=4229886755
rootdir: /home/batman/openpilot
configfile: pyproject.toml
plugins: timeout-2.3.1, xdist-3.6.1, cov-5.0.0, mock-3.14.0, asyncio-0.24.0, sugar-1.0.0, randomly-3.15.0, cpp-2.6.0, flaky-3.8.1, hypothesis-6.47.5, anyio-4.4.0, subtests-0.13.1, repeat-0.9.3
asyncio: mode=Mode.STRICT, default_loop_scope=function
16 workers [211 items]    collecting ... 

 selfdrive/car/tests/test_car_interfaces.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 60% ██████    
                                            ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                         100% ██████████
================================================================================= slowest 10 durations ==================================================================================
1.27s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_001_ACURA_RDX
1.22s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_032_FORD_MUSTANG_MACH_E_MK1
1.20s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_027_FORD_EXPLORER_MK6
1.17s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_026_FORD_ESCAPE_MK4
1.16s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_028_FORD_FOCUS_MK4
1.16s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_042_GENESIS_GV80
1.15s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_017_CHEVROLET_VOLT
1.14s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_047_HONDA_CIVIC_2022
1.12s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_025_FORD_BRONCO_SPORT_MK1
1.12s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_202_VOLKSWAGEN_POLO_MK6

Results (14.30s):
     211 passed
batman@workstation-shane:~/openpilot$

So to achieve the original goal, we need to see about a ~5-6x speedup still.

[c-gamble]

Getting this result on a AMD Ryzen Threadripper PRO 3955WX 16-Cores:

I ran on m3 MBP—appreciate the clarification!

[JIBSIL]

Hey guys,

This bounty appears open under Projects so I attempted it. I believe with a few changes I have improved it to your specifications without making the tests less effective (about a 8x improvement on my machine, i7-7700k). I'll open a PR in a bit, can you guys lock the bounty if my PR is satisfactory? First PR I've done on this repository by the way, so please let me know if there's anything I am missing.

[sshane]

You can put up a PR and we will review it.

[blackoutnet]

Hey guys, please have a look: https://github.com/commaai/openpilot/pull/35183

[zarisss]

hey, ive worked on this bounty and got the following results

============================= test session starts ============================== platform linux -- Python 3.12.3, pytest-8.3.5, pluggy-1.6.0 Using --randomly-seed=1784375198 rootdir: /home/sharish/openpilot configfile: pyproject.toml plugins: cpp-2.6.0, subtests-0.14.1, repeat-0.9.4, xdist-3.6.1, asyncio-0.26.0, randomly-3.16.0, hypothesis-6.47.5, timeout-2.4.0, cov-6.1.1, mock-3.14.0 asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function created: 8/8 workers 8 workers [226 items]

........................................................................ [ 31%] ........................................................................ [ 63%] ........................................................................ [ 95%] .......... [100%] ============================= slowest 10 durations ============================= 0.58s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_034_FORD_MAVERICK_MK1 0.58s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_032_FORD_F_150_LIGHTNING_MK1 0.58s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_031_FORD_FOCUS_MK4 0.54s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_033_FORD_F_150_MK14 0.53s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_035_FORD_MUSTANG_MACH_E_MK1 0.52s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_036_FORD_RANGER_MK2 0.50s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_029_FORD_ESCAPE_MK4_5 0.50s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_027_FORD_BRONCO_SPORT_MK1 0.48s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_028_FORD_ESCAPE_MK4 0.47s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_030_FORD_EXPLORER_MK6 ============================= 226 passed in 11.55s =============================

[iam-saju]

is it still available

[hanzhi227]

There's 227 tests, so if each test average is <0.2s then the total test time <45.4s which it is already at ~22s for me. Simplest way to reduce the test time so that the highest test times are around 0.2s is to just reduce the MAX_EXAMPLES from 60 -> 15, but even at 60, the test times is averaging <0.1s per test.

================================================================================================================ test session starts ================================================================================================================
platform darwin -- Python 3.11.4, pytest-8.4.1, pluggy-1.6.0
Using --randomly-seed=1967957443
rootdir: /Users/hanzh/Dev/Projects/openpilot
configfile: pyproject.toml
plugins: randomly-3.16.0, cpp-2.6.0, repeat-0.9.4, timeout-2.4.0, mock-3.14.1, hypothesis-6.47.5, xdist-3.7.1.dev3+g909e97b, subtests-0.14.2, asyncio-1.0.0
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
8 workers [227 items]   
...................................................................................................................................................................................................................................           [100%]
=============================================================================================================== slowest 10 durations ================================================================================================================
0.28s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_028_FORD_ESCAPE_MK4
0.28s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_029_FORD_ESCAPE_MK4_5
0.28s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_035_FORD_MAVERICK_MK1
0.26s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_031_FORD_EXPLORER_MK6
0.26s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_032_FORD_FOCUS_MK4
0.25s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_033_FORD_F_150_LIGHTNING_MK1
0.24s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_030_FORD_EXPEDITION_MK4
0.23s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_027_FORD_BRONCO_SPORT_MK1
0.18s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_000_ACURA_ILX
0.18s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_037_FORD_RANGER_MK2
================================================================================================================ 227 passed in 6.42s ================================================================================================================```

[ReyNeill]

Participated on this bounty, opened 2 related PR, no apparent compromises, better than requested results

opendbc: https://github.com/commaai/openpilot/pull/36560 openpilot: https://github.com/commaai/openpilot/pull/36560 (expected job error dependent on opendbc PR)

[logesh45]

I was able to speed this up with minimal changes to opendbc submodule.

My local run shows 6.52ss before and 0.79s slowest test after the change

https://github.com/commaai/opendbc/pull/2888

=================================================================== slowest 10 durations ==================================================================== 0.79s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_037_FORD_MAVERICK_MK1 0.76s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_099_HYUNDAI_NEXO_1ST_GEN 0.76s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_038_FORD_MUSTANG_MACH_E_MK1 0.72s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_104_HYUNDAI_SANTA_FE_HEV_2022 0.72s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_035_FORD_F_150_LIGHTNING_MK1 0.71s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_057_HONDA_CIVIC_BOSCH 0.71s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_031_FORD_ESCAPE_MK4_5 0.70s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_033_FORD_EXPLORER_MK6 0.70s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_030_FORD_ESCAPE_MK4 0.70s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_032_FORD_EXPEDITION_MK4 =================================================================== 236 passed in 15.78s ====================================================================

My before change test time with M1Pro was 38.84s with slowest test at 6.52s. An 8x improvement.

[logesh45]

The above pr needs to be merged in opendbc repo to see performance improvements in the main openpilot repo. Some eyes here when you get a chance please.

[TheSecurityDev]

The above pr needs to be merged in opendbc repo to see performance improvements in the main openpilot repo. Some eyes here when you get a chance please.

I believe you can test this by forking both openpilot and opendbc, and configuring your openpilot fork to point to your opendbc fork.

[logesh45]

will those tests still run? I have no changes for openpilot repo.

[TheSecurityDev]

will those tests still run? I have no changes for openpilot repo.

I think it will

[ReyNeill]

will those tests still run? I have no changes for openpilot repo.

I think it will

Could I get a feedback emoji rating the work of my opened PRs? I did tested with a fork of both openpilot and opendbc

https://github.com/commaai/openpilot/pull/36560 https://github.com/commaai/openpilot/pull/36560

[logesh45]

The above pr needs to be merged in opendbc repo to see performance improvements in the main openpilot repo. Some eyes here when you get a chance please.

I believe you can test this by forking both openpilot and opendbc, and configuring your openpilot fork to point to your opendbc fork.

I did create a PR #36600 and it's tiny bit better when it runs on GitHub actions.

I think it's already 0.2s per test there even before the changes here's an example completely unrelated PR from this morning #36599. the unit tests run in 1m 22s before and after my change, it is 1m 20s.

[logesh45]

@adeebshihadeh @sshane Can you run this branch on your machine and post results? there is only ~2s improvement in the CI though. Individual car tests are now 7x faster but still the total test time in CI hasn't;t come down since it was already faster in the CI. I compared it with a previous PR results.

[logesh45]

https://github.com/commaai/opendbc/pull/2888#pullrequestreview-3490056013

@adeebshihadeh @sshane @jyoung8607 can this be marked as solved now? I'm looking for an interview slot.

[dzinesco]

🐕 Dog the Bounty Hunter has claimed this bounty!

PR: https://github.com/commaai/openpilot/pull/36751