sdk-ng icon indicating copy to clipboard operation
sdk-ng copied to clipboard

Add QA test workflow

Open stephanosio opened this issue 3 years ago • 0 comments
trafficstars

Add a workflow for QA (quality assurance) testing.

The purpose of the QA workflow is to automate the process of building and running the Zephyr kernel tests on all runnable targets, in order to verify that both the SDK toolchains and the emulation tools (e.g. QEMU) are functional and free of bugs/breakages.

The QA workflow is different from the main CI workflow, which also tests the toolchains and host tools, in that the main CI workflow focuses on smoke testing by building and running only the most basic Zephyr tests on the major targets, whereas the QA workflow focuses on functional testing by building and running the full Zephyr kernel and core OS tests on all runnable emulation targets.

The QA workflow shall:

  • be a GitHub Actions workflow that is manually dispatchable through the workflow UI in the "Actions" tab.
  • allow specifying the target SDK ref and Zephyr ref.
  • build and run, on all runnable targets, the Zephyr kernel and core OS tests, which include but are not limited to:
    • Kernel tests (tests/kernel)
    • Arch tests (tests/arch)
    • C Library tests (tests/lib/newlib)
    • C++ Library tests (tests/subsys/cpp/libcxx)
  • publish the test report

Sample test command line:

scripts/twister -v -N -M --retry-failed 5 -T tests/kernel --filter runnable

stephanosio avatar Mar 18 '22 01:03 stephanosio