ardupilot
ardupilot copied to clipboard
ArduPilot
Reduce Autotests for Granular Pull Requests
Feature request
Is your feature request related to a problem? Please describe. The autotest functionality is useful, but tests can take quite a while to complete, and get backed up for several hours (or days) when there are multiple PRs in a short period. The backup can be frustrating when it is clear that several of the tests aren't relevant to the PRs being tested.
Describe the solution you'd like Ideally an automated way to make each test only run when it's relevant for it to do so. If a test is not testing any changes, it should not be run.
A key example is that a PR that modifies only files in a vehicle-specific sub-folder (e.g. ArduCopter) has no need to run tests for other vehicles (e.g. ArduPlane). I presume there are other relevant cases too (e.g. many Tools modifications likely don't affect most/any of the vehicle tests, and some firmwares may not support some hardware types so could perhaps avoid some size tests or something), but I don't know enough about the testing setup to give clear examples that would be practically significant.
Describe alternatives you've considered From a practical standpoint, it may be simpler for each test to do its own detection of whether relevant files have been changed, and if not finish early. That's less ideal than the "don't run at all" approach because it keeps the clutter from running each individual action, and it could be confusing/misleading to return a pass or fail for a test that was not actually run (not sure if there's a "not run" output option or something), but it should still provide most of the same performance improvements.
Platform [x] All [ ] AntennaTracker [ ] Copter [ ] Plane [ ] Rover [ ] Submarine
Additional context Add any other context or screenshots about the feature request here.
I've found the relevant CI docs for path inclusion/exclusion.
The PR Willian linked to added path exclusion to avoid testing vehicles other than the vehicle folder being modified :-)
Changes to Tools and whatnot still run all tests (AFAIK), which would be nice to address.
@khancyr did a whole lot of work to not run tests that we don't need to. I think we can close this, @ES-Alexander ?
