Checkin test script fails when the ctest exe is anything other than ctest

[bmpersc]

I received a bug report from a user that when they ran the checkin test script the testing step failed because it couldn't find ctest. They were confused by this because they had never had any issues testing manually running "make test" on the machine. It turns out that on his machine ctest is named ctest28. It appears that cmake knows this and generates makefiles that use the ctest28 name. I believe this can be easily resolved by having the checkin test script run "make test" instead of ctest directly. There is a cmake variable that stores the name of the ctest exe, but I'm not sure if it is easily accessible outside of cmake.

[bartlettroscoe]

Brent,

Does make test allow you to run multiple tests at the same time like with ctest -j<N>? If not, there is no way we want to use make test. Also, note that we are also going to need to support Ninja which will not be invoked with make so we can't assume make test anyway. The checkin-test.py script will need to be updated to support ninja. To support ninja and other generators targets, we can't hard-code make at all. I am not sure how to address that issue just yet.

Instead, why not just just add an option --with-ctest like we support for --with-cmake (see documentation)? If we did that, the user could just provide --with-ctest=ctest28 and that would take care of it. Also, we can have the checkin-test.py script look for ctest up front and then bomb out if it can't find it (with which ctest). This is what it does for eg at the very beginning (with will be just raw git once I can get to #15).

But of course, if I were on a machine like this, I would just add aliases like:

alias cmake=cmake28
alias ctest=ctest28

and be done with it. But perhaps that is too easy to suggest for this user?

[bmpersc]

Ross, The make test target does, in fact, support multiple tests at the same time, but it is not obvious how to do it. It requires setting ARGS to the right value, eg. make test ARGS="-j 12".

I haven't used ninja so I do not know if cmake's ninja generator supports a test target like the make generator does. However, this point is well taken. I don't know if there are plans for allowing the checkin test script to allow the VS generator, but I know it doesn't support a test target in the same manner as the make generator does. It would be short sighted to add in a different porting error to fix this issue.

A --with-ctest option would actually be a reasonable solution to this. It wouldn't help with the complexity complaint that the above issue was initially buried in when the user came to me, but that is a separate issue. This does still force the user to know about this renaming, but that isn't an entirely unreasonable requirement.

Using aliases here is down right absurd and I hope that was a joke suggestion. That is akin to telling users that you can't change the compilers with an option, instead if they want to use different compilers they must alias cc, CC and f77 to the respective compilers that they want to use. GNU compilers frequently come as gcc47 or gcc34 and no one seems to mind, I don't know why cmake tools should be treated differently especially when it seems cmake knows how to handle this. FWIW the users system did not have a cmake28, the cmake name was unmodified, but somehow knew about the modified ctest name. I didn't look into the matter too deeply to figure out why, how or if other cmake tools were named in this fashion. However, since cmake seems to support this kind of configuration it seems reasonable that any tools working with cmake ought to let cmake do the heavy lifting on this and follow its lead on what the individual tools are called.

[bmpersc]

There appears to be a fairly simple way to discover what the name of ctest is by using an option to cmake. On linux and mac "cmake --system-information |grep CMAKE_CTEST_COMMAND" should report the full path to the ctest exe that it expects to use. I'm sure there is a much more portable way to do this just by scraping the output and using python's built in regex module. But that is the simple solution to finding the ctest exe name that I found in like 2 minutes of looking through cmake --help's output. I'm still all for having an option to allow a user to override the ctest, but I think using that value as the default instead of just "ctest" would be the best solution here.

[bartlettroscoe]

Okay, I have the solution. The solution is to grab the ctest command out of the CMakeCache.txt file. It seems that CMake records its sister CTest executable path in the variable CMAKE_CTEST_COMMAND. For example, on my system this is:

CMAKE_CTEST_COMMAND:INTERNAL=/projects/vera/common_tools/cmake-2.8.11/bin/ctest

I know that Kitware documentation says not to assume the structure of the CMakeCache.txt file but I have never seen it change. It would be pretty easy to write Python code that could extract any variables out of the CMakeCache.txt file you wanted. (That function itself could be pretty handy actually.) This would require implementing a Python function that can read variables out of a CMakeCache.txt (and unit testing it) and then adding this function call to code in CheckinTest.py to get the path to ctest before calling it.

How urgent is this? There are a lot of higher priority things the checkin-test.py script needs in my opinion, as documented here. However, if someone wants to give implementing this a try and submitting a pull request, I would review it and push it if it was up to snuff (perhaps after some refactoring).

[bartlettroscoe]

On linux and mac "cmake --system-information |grep CMAKE_CTEST_COMMAND" should report the full path to the ctest exe that it expects to use.

That is a better solution. Since the CTest you use should always be matched to the CMake you use, we should not even bother with a --with-ctest argument.

The hardest part of making this change will be updating the automated tests.