Customize command for running test binaries

[simonratner]

It is useful to specify a custom executor script for running tests as part of the newt test command. This could be specified as a key in pkg.yml for unittest packages, analogous to bsp.downloadscript and bsp.debugscript which exist for bsp packages.

This script will be executed by newt test with the (relative) path to the test binary as a parameter, in place of the binary itself. The default should be to exec the binary directly, as it is now.

See: https://github.com/apache/mynewt-newt/blob/master/newt/builder/selftest.go#L151

Motivation

WSL on Win10, as well as macOS Catalina, are exclusively 64-bit and are not able to run the test ELFs. While both of these systems are capable of building the unit tests with crosstools, running the resulting binary requires a VM/container with a 32-bit-capable OS. May also be useful for non-WSL Windows environments.

Workaround

Run newt test to build the test package, which will fail with an ELF format error.

$ newt test test_pkg
Testing package test_pkg
...
Executing test: bin/targets/unittest/test_pkg/app/test_pkg/test_pkg.elf
Test failure (test_pkg):
fork/exec bin/targets/unittest/test_pkg/app/test_pkg/test_pkg.elf: exec format error
Error: Test failure(s):
Passed tests: []
Failed tests: [test_pkg]

Then manually execute the built binary in a supported environment:

docker run -e NEWT_USER=$(id -u) -e NEWT_GROUP=$(id -g) -e NEWT_HOST=$(uname) \
    --rm --privileged -v "$(pwd):/workspace" -w /workspace \
    mynewt/newt:latest \
    bin/targets/unittest/test_pkg/app/test_pkg/test_pkg.elf

[ccollins476ad]

Thanks @simonratner, that is an interesting idea.

I'm wondering if the test package is the best place for the custom command to be specified. Ideally, a unit test would continue to work in 32-bit linux as before, and the custom command would only be used in other environments. If the custom command is specified in the test package, that would change how the test gets run in all environments.

newt test could accept a command line argument that specifies the custom command. What do you think of that idea?

[simonratner]

I think command line is fine, something like --executor=<script-accepting-filename-as-param>.

By the way, WSL2 does run ELF32 binaries directly (since it is a real Linux kernel). I modified the description to reflect that.