Continuous integration tests vs nightly tests

[lxmota]

@ibaned Discussing with @calleman21 and @jwfoulk we think it would be useful to have two categories of tests:

• Continuous integration: tests that run for less than 5 seconds that will be run before any check ins. These tests would be part of the main repo (sort of like small tests now).
• Nightly tests: longer than 5-second runs that test more thoroughly other features. No need to run these before check ins. These tests would be part of an optional testing repo (sort of like large tests now).

We think that this break down of the tests could replace the current division into large and small tests. It would also be more useful from a developer's point of view and discourage the practice of checking in commits without running any tests.

This is related to https://github.com/gahansen/Albany/issues/39

Adding @jtostie to the thread.

[ibaned]

So, the fundamental difference is disk space versus runtime. The current small/large distinction is there to minimize the disk space consumed by the small test files. This is not inherently correlated to the runtime of the tests, although in practice the two are usually somewhat connected. I am fine with having both restrictions on the "small" category, but I don't think we can lift the disk space restriction.

[lxmota]

Yes, we were just discussing that. One idea would be to leave the small/large division as is as part of the nightly tests and select from those which tests would be part of the continuous integration suite.

[ibaned]

I think the fast/slow distinction can be made at the CMake level instead of the filesystem level. You can probably introduce some kind of ENABLE_SLOW_TESTS and start putting if(ENABLE_SLOW_TESTS) around certain tests which you think are taking too long. I think we should be more careful with backwards compatibility this time, and make ENABLE_SLOW_TESTS default to ON until we can get all the dashboard builds to explicitly turn it ON, otherwise many tests will disappear from CDash.

[calleman21]

@ibaned First, I want to say thanks for the work you did to separate the tests. I think that this is an important effort, so thank you for taking the initial steps. I do think that we need to continue the discussion about how this is done based on our experiences developing subsequent to this significant change.

I agree that the issues of disk space and runtime are not one-to-one, but I am arguing that runtime is a more appropriate metric. I understand the desire to manage the size of the repo, but I think that concerns about disk space should only be addressed when doing so does not negatively impact other development concerns.

I want to bring up the following three items, from my personal point-of view: 1) I would guess that in practice, fast tests are small tests. Albany takes seconds to read in large meshes, so I believe that requiring tests to run fast includes requiring tests to be small. Thus, I think this debate is largely philosophical (but that doesn't mean it's not important); 2) I don't think that it is really appropriate to use disk space to divide tests. If testing some important functionality in Albany requires a particular problem, then that problem should run regularly, regardless of disk space concerns. If the runtime is large, regularly should be nightly, and if it can be run quickly, then regularly should be prior to each push. I suspect that it will almost never be the case that we really require a 'large' test to be run before each push; 3) Personally, I don't really like the idea of using CMake as described above. I would prefer to divide the tests into appropriate directories, as you did, rather than invoking CMake functionality to accomplish this. The directory solution just seems simpler to me.

Thoughts?

[ibaned]

So, there is a lot of confusion generated about why I want to separate things by disk space. If it were up to me, Albany would be two separate repositories: one only source code and the second only input files, including any and all tests. The reason for this is that not all use cases require the test files. We often want to simply check out the code on some new machine like a supercomputer, compile it once, and run some large cases. Either we have users now who are not developers, or we would like to in the future.

Right now the Albany repository is bigger even than the Trilinos repository, even though it is a tiny fraction of the number of lines of code, because the Albany repository is almost all test input files (30MB of source code, 800MB of tests). That to me is a really really bad use of a Git source repository. We can do a lot better for users who don't need to carry around all those input files because they will never run the tests. I was once asked to snail mail a thumb drive with the Trilinos repository contents so that I could get it onto a cluster which had a restricted Internet connection. These are the kinds of situations we can avoid getting ourselves into if there is a way to clone just Albany's source code and no more.

The only reason there is any separation of the test files at all based on size is that @agsalin insisted that some test files be kept with the source code repository.

My intent would be that all developers should clone the input files for tests they are required to run, regardless of size, and that the dashboard scripts clone all the input files, including the large/slow ones.

As for how to separate tests, after doing it both through local CMake logic and with sub-directories, I'm pretty sure CMake is cleaner. You'll notice the subdirectories have mirror images of the directory sub-trees, and a lot of CMake logic is duplicated and has to be juggled between the two halves. The reason I did subdirectories is because tests/large is supposed to become its own Git repository, so that needs to be a separate directory.

[bartgol]

I think we need both filters, space and time. The two are however orthogonal: filtering the tests by input files size is useful to segregate large files in a different git repo, and keep the main albany repo small; on the other hand, filtering the tests by run time is important for those developers that produce several commits per week, to allow them to easily and quickly test/push the new modification they implemented, without having to wait a long time (perhaps a day, waiting of nightly build tests). In this sense, I would not look at this as a dichotomy of one vs the other, as they serve different goals in different ways.

I think I like the solution proposed by @ibaned . Once the large test folder becomes its own repo, we will be left with "small" size tests, and we will have to make them faster. Ideally, all the tests should take little time, but if that's not possible for some reason, then having a cmake option Albany_ENABLE_SLOW_TESTS or Albany_ENABLE_EXTENSIVE_TESTING would allow us to filter the "slow" tests out, and allow fast testing for continuous integration.