capstone icon indicating copy to clipboard operation
capstone copied to clipboard
verified publisher icon capstone-engine

Modern Testing

Open Rot127 opened this issue 1 year ago • 2 comments

Your checklist for this pull request

  • [ ] I've documented or updated the documentation of every API function and struct this PR changes.
  • [x] I've added tests that prove my fix is effective or that my feature works (if possible)

Detailed description

Supersedes https://github.com/capstone-engine/capstone/pull/2456

Modernize testing.

The new unit testing is build around yaml files. They can be consumed by the C test engine (cstest) or any bindings easily.

Statistics

The new cstest logs technically two statistics. One by cmocka and one by cstest. The reason is:

  1. We might want to group tests in the future (by arch or file for example) so we need a global statistics.
  2. Errors of incorrect yaml parsing are not tracked by cmocka.
  3. Also, cmocka doesn't provide the detailed statistics to us. So we cannot print a nice summary at the end.

Changes

  • New cstest C-implementation
  • New py_cstest Python-implementation
  • Moves almost all tests to yaml files.
  • Old tests which can't be moved (because they are effectively integration tests) are kept.
  • Add numerous test jobs in the workflow
  • Adds several CMake test targets
  • Add LoongArch Python bindings
  • Several bug fixes discovered during testing.

Current test coverage of Auto-Sync <ARCH>Mapping.c files by running cstest test/:

file coverage covered
arch/LoongArch/LoongArchMapping.c 67.29% 144 / 214
arch/PowerPC/PPCMapping.c 72.78% 254 / 349
arch/AArch64/AArch64Mapping.c 60.47% 918 / 1518
arch/TriCore/TriCoreMapping.c 45.00% 45 / 100
arch/Alpha/AlphaMapping.c 86.08% 68 / 79
arch/ARM/ARMMapping.c 61.68% 792 / 1284
arch/HPPA/HPPAMapping.c 29.85% 20 / 67

After merge

  • [ ] Add windows build to CI

Test plan

Added and all green

Closing issues

closes https://github.com/capstone-engine/capstone/issues/1984 closes https://github.com/capstone-engine/capstone/issues/2405 closes https://github.com/capstone-engine/capstone/issues/2383 closes https://github.com/capstone-engine/capstone/issues/2383

Rot127 avatar Aug 25 '24 10:08 Rot127

I am aware that the Auto-Sync test fails. I have no idea why (it works fine locally) but fix it later. Just wanted this ready for review.

The clang-tidy issue is a false positive. But I cannot for the sake of it create a minimal working example to report the bug.

I am not sure if I find the time to try it again. But either we force merge or silence it.

Rot127 avatar Aug 25 '24 10:08 Rot127

@kabeor I fixed the Auto-Sync workflow. Please take another look.

Rot127 avatar Aug 28 '24 06:08 Rot127