tests for ensuring correct installation with tests in VM or something else

[bulwahn]

The existing test script checks that all packages can be build. However, it does not ensure that the packages can run in an image. We must also provide tests that check this property for all the packages.

First, we should discuss how one could implement that in general.

[KristofRobot]

On how to test whether packages install correctly -> this can easily be tested by adding the package to the IMAGE_INSTALL parameter (either in a custom package, or in your local.conf), e.g.

IMAGE_INSTALL += "packagegroup-ros-world"

core-image-ros-world.bb has been added to provide this capability for the full ros-world package - i.e. to test whether packagegroup-ros-world install succesfully, just run:

bitbake core-image-ros-world.bb

[KristofRobot]

On how to test whether packages run correctly - I think it probably makes sense to first try to understand how to do this manually, before automating.

Personally, I haven't found a good approach here yet, and would be interested to hear what others are doing.

Currently I take following steps: _One time setup_ A. Build core-image-ros-roscore for qemux86 B. Run image in qemux86 and do some one time configuration (exporting ROS_MASTER, configure opkg feed server, ...) This is now a base image with a minimal ros stack, ready for future tests.

_Test whether package runs correctly_ I'm currently exploring following procedure to check for the necessary RDEPENDS

1. Run the base image prepared above in snapshot mode, so that all changes are removed when it shuts down, ready for next testing
2. Bitbake the package to test, and opkg install in this snapshot (I find this easiest by using feedserver)
3. Try to run the package.
4. Fix any run dependencies uncovered, and repeat

I find (3) typically the most challenging step - figuring out a good test command. I wonder whether we can somehow run ROS unit/integration tests here ([0],[1], [2], [3]) ? Anyone experience with that?

Once we have this manual procedure defined, it shouldn't be too hard to automate this, e.g. by leveraging the OpenEmbedded TEST_IMAGE functionality ([4], [5]).

Thoughts?

[0] http://wiki.ros.org/UnitTesting [1] http://wiki.ros.org/gtest [2] http://wiki.ros.org/rostest [3] http://wiki.ros.org/regression_tests [4] http://cgit.openembedded.org/openembedded-core/tree/meta/classes/testimage.bbclass [5] http://cgit.openembedded.org/openembedded-core/tree/meta/classes/testimage-auto.bbclass

[bulwahn]

I agree with your approach and your ideas.

To get the ROS tests running that are shipped with the common ROS packages, we have to look how catkin, gtest, and cmake set up the compilation and the execution of the tests. I believe currently compiling and running the tests is glued together in the make_tests build target, and hence, the tests are tried to be executed just after the compilation on the same machine. To split the two tasks, compiling and running, we have to adjust the whole catkin/cmake tool chain, e.g., compile the tests, install the tests, install a proper test script and hence allow testing on the target. It would be best if this is possible to be non-invasive on the CMakeLists of all the ROS packages. Otherwise, this will need a lot of adaptations for all the ROS packages.

Do you want start with an initial investigation, which tools and files set up the testing in ROS packages and which things have to be changed for allowing to cross-compile the tests?

[yangyy11]

ok