meshtastic
[Feature Request]: Hardware in the loop testing
Platform
NRF52, ESP32
Description
It seems like it would be a good idea to do something around hardware-in-the-loop testing, to enable new releases to have some amount of testing on real hardware before they go out into the wild. I'm setting out a plan for how I might tackle this, but I'm open to feedback and modification. It may be worth breaking down into multiple issues. It's also worth noting that in the long term, this will need some community support, in the form of volunteers with different boards and configurations willing to run a github local runner.
If you don't believe this adds value, or you don't think the community is willing and able to support this, please feel free to close this.
- [ ] Read/understand the current CI/CD structure.
- [ ] Figure out how to run a github local runner that has access to hardware.
- [ ] Create a test in github CI that can call the local runner and flash firmware to a board.
- [ ] Create a method to automate first-time configuration.
- [ ] Parameterize appropriately so that it can scale out to different runners with different boards.
- [ ] Document the heck out of this so that new people can follow a script to add new boards to the fleet.
- [ ] Onboard additional volunteers based on the above.
- [ ] Investigate extending the tests to cover more functionality. For example, a really useful test would be to flash two boards with new firmware, start the mesh, and send a message from each and confirm it was received by the other.
I’ve done this kind of testing professionally.
I foresee the hardest part of this endeavor being finding someone willing and able to set up the device lab in their home. I do not have the money or space, personally, but am willing to help guide and troubleshoot such an endeavor.
One thing to consider for something like this is the Linaro LAVA project. It’s a bit overly complicated IMO, but it is designed to handle firmware flashing and device lockout as part of CI. It, of course, can also be done via tagging your hosted build agents as well.
