catkin run_tests should run only one job at a time (--jobs=1) for specified packages

[kopp]

This is rather a question than a bug/issue/...: When running catkin run_tests, some tests require to run only one test at a time (particularly when they launch nodes and run a roscore). There is a second parameter (called force_single_threaded_when_running_tests) to handle_make_arguments(..) (in catkin_tools/argument_parsing.py) to do just that, i.e. to enable --jobs=1 if catkin run_tests is executed. It defaults to False. In the only two places I found handle_make_arguments being used (catkin_tools/jobs/cmake.py and catkin_tools/jobs/catkin.py), only one parameter is passed to that function, though, i.e. the feature is never enabled.

So my question: What is/was the intended mechanism to enable this feature?

(I run catkin 0.4.2 with python 2.7.6 on Ubuntu 14.04 with ROS Jade, but I don't see how that affects the issue; it seems to be the same in catkin 4f5fa9dc318f224.)

[wjwwood]

@kopp what does 4f5fa9d have to do with this? Did you mean to link to a ros/catkin commit?

[wjwwood]

Historically we did limit tests to -j1, but I seem to remember that getting addressed in rostest so that it wasn't a problem anymore. It's still very possible that tests could interfere with one another, but since rostest started using unique rosmaster ports it was no longer the case that parallel tests always interfered with each other. I'd have to do some GitHub archaeology to find it, but I believe that's what happened. I think the force_single_threaded_when_running_tests was copied from ros/catkin which was used in the catkin_make and catkin_make_isolated tools.

Basically, at some point in the past it was true that -j1 was essentially required for test running, but that I think it is no longer mandatory, and so should not be the default option. However, I could be wrong about that in which case we should change the run_tests verb so that it does not run them in parallel by default.

You can pass -j1 and/or -p1 after run_tests to ensure you're not running tests in parallel if that is a problem for you.

[kopp]

@wjwwood Sorry for the confusion: I wanted to say that I tested with that version; it has nothing to do with that particular commit.

In our case, we have unit tests from one package interfering with each other and so what I can do on the command line is to set -j1 when running catkin run_tests for that particular package. When we run all our tests on a Jenkins, we do not want to run all tests with -j1, though. So I would like to mark some packages to be problematic in the sense that for that package -j1 should be enabled. As far as I understand, there is no mechanism for that right now and there are no plans for something like that.

[wjwwood]

I believe https://github.com/catkin/catkin_tools/issues/205 could be used as you've described.

[kopp]

I would rather like to put it into the CMakeLists.txt: When I know that tests must not run in parallel, I don't want to have to configure that in catkin but keep it "close to the test". I don't know, though, whether it's possible to change the number of jobs within the CMakeLists.txt file. If not, you're right and #205 looks like the way to go by setting -j1 as make argument for that specific package.

[jhoare]

@kopp Could you configure your tests to run serially: http://stackoverflow.com/questions/29983138/force-serial-execution-for-specific-targets-in-cmake

[Vuwij]

Hi, I have the same issue. The problem is that the tests will definitely interfere with each other (both POSTING and GETTING from a common web server. And I want to run the tests sequentially so that one .test file starts running with the maximum number of cores and then the second one as well Something like what @jhoare posted would be ideal