Improve pytest collection time to <1s

[sshane]

Currently it takes ~6-7s with these plugins:

platform linux -- Python 3.11.4, pytest-8.2.1, pluggy-1.5.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Using --randomly-seed=3123610973
rootdir: /home/batman/openpilot
configfile: pyproject.toml
testpaths: tools/, common/, system/, selfdrive/
plugins: anyio-4.3.0, timeout-2.3.1, xdist-3.6.1, cpp-2.5.0, cov-5.0.0, mock-3.14.0, forked-1.6.0, dash-2.11.1, benchmark-4.0.0, randomly-3.15.0, subtests-0.12.1, flaky-3.8.1, hypothesis-6.47.5, asyncio-0.23.7, nbmake-1.5.3, repeat-0.9.3
asyncio: mode=Mode.STRICT

============================================================================= 4011 tests collected in 6.46s ==============================================================================

real    0m8.935s
user    0m10.833s
sys     0m2.163s

Verification:

Run time pytest --collect-only in the root openpilot directory and post both the real time and the collection time as reported by pytest (=== 4011 tests collected in 6.61s ===). Include the times before and after your PR.

[BBBmau]

should the time improvement come from not making changes to the testpath?

You can easily improve by being explicit in testpaths. Doesn't seem like that's the point of this bounty if it's that easy.

[sshane]

pytest selfdrive/car/tests --co takes 1.5s on its own.

[renxida]

============ 4172 tests collected in 2.05s ============

running pytest --co today gives the above output.

Perhaps it's time to close this and give the bounty to whoever got this here.

(just tried again. when i delete the cache, it takes 3.76s)

[Shaikimram]

hey @sshane, is anyone working on this issue, I would like to take it up.

[Ryan-Amirthan]

I ran the collection command multiple times and consistently achieved collection times under/around 2 seconds:

time pytest --collect-only 
================================================================== 2885 tests collected in 1.93s ===================================================================
pytest --collect-only  1.75s user 0.34s system 87% cpu 2.380 total
================================================================== 2885 tests collected in 2.07s ===================================================================
pytest --collect-only  1.72s user 0.41s system 76% cpu 2.781 total
================================================================== 2885 tests collected in 2.03s ===================================================================
pytest --collect-only  1.72s user 0.39s system 79% cpu 2.665 total
================================================================== 2885 tests collected in 1.80s ===================================================================
pytest --collect-only  1.71s user 0.34s system 87% cpu 2.340 total
================================================================== 2885 tests collected in 1.98s ===================================================================
pytest --collect-only  1.70s user 0.38s system 79% cpu 2.602 total

Based on these 5 tests, they are averaging 1.962s. Even after clearing the cache between runs, the collection time remained under 2 seconds.

My environment (if that helps): MacOS Sequoia 15.3.2 M3 Python 3.12.8 pytest 8.3.5 5d1816e2b (latest commit)

this issue seems to have been resolved through organic improvements to the codebase or pytest itself.

I would recommend closing this issue as the performance target seems to be met without requiring additional changes.

[adeebshihadeh]

It does seem to have improved, here's what I get on my machine.

======================================================================= 2885 tests collected in 2.07s ========================================================================

real	0m2.722s
user	0m7.156s
sys	0m0.334s

Moving the target to 1s!

[raf-fonseca]

@adeebshihadeh or @sshane is anybody working on this? I would like to take it up

[NoahCalabro]

@raf-fonseca Still working on this? I may want to take it up in case you aren't anymore.

[sgtBurns7]

I am able to run the tests in 1.18~1.19 s constantly is the target still 1s?

[HeathNaylor]

@sshane I'm interested in looking at this but I am seeing 3019 tests vs your 4011 tests, I am currently on the master branch. Is there a specific branch you are targeting? Also what about hardware differences, do we have a hardware spec we are targeting through Github Actions for example?

=============================================================================== 3019 tests collected in 2.80s ================================================================================

________________________________________________________
Executed in    4.28 secs    fish           external
   usr time    4.35 secs    0.17 millis    4.35 secs
   sys time    0.35 secs    1.06 millis    0.35 secs