ci: run test in parallel

[hfudev]

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[hfudev]

I tried locally with a simple example and pytest-embedded should work with pytest-xdist, if there's no duplicated target-port got assigned.

For example

import pytest

from pytest_embedded_idf.dut import IdfDut


@pytest.mark.parametrize('app_path', [
    '/home/fuhanxi/esp/esp-idf/examples/get-started/hello_world',
], indirect=True)
@pytest.mark.parametrize('target,build_dir', [  # different targets, auto-detection still works
    ('esp32c6', 'build_esp32c6_default'),
    ('esp32', 'build_esp32_default'),
], indirect=True)
def test_hello_world(
    dut: IdfDut,
) -> None:
    dut.expect('Hello world!')

and here's a benchmark with xdist and without xdist, with two cases

hyperfine "pytest -n 2 test.py --embedded-services esp,idf" "pytest test.py --embedded-services esp,idf"
Benchmark 1: pytest -n 2 test.py --embedded-services esp,idf
  Time (mean ± σ):      5.121 s ±  0.346 s    [User: 0.697 s, System: 0.168 s]
  Range (min … max):    4.781 s …  6.056 s    10 runs
 
Benchmark 2: pytest test.py --embedded-services esp,idf
  Time (mean ± σ):      9.036 s ±  0.675 s    [User: 0.327 s, System: 0.093 s]
  Range (min … max):    8.542 s … 10.295 s    10 runs
 
Summary
  pytest -n 2 test.py --embedded-services esp,idf ran
    1.76 ± 0.18 times faster than pytest test.py --embedded-services esp,idf

known limitation of using pytest-xdist

• can't trace serial output in job log. ref

cc @igrr @espzav

[github-actions[bot]]

Test Results

 15 files  15 suites   3m 57s ⏱️  45 tests 14 ✅  23 💤 0 ❌ 8 🔥 330 runs  14 ✅ 308 💤 0 ❌ 8 🔥

For more details on these errors, see this check.

Results for commit 3f83e562.

[espzav]

@hfudev Thank you very much for this solution! It seems that it can work, when we will select only one example for tests for each board. Otherwise, we will make some process to synchronize it. I will try it soon with these changes :-)

[espzav]

Solved in #564 --> closing Thank you!