Reorganize tests

[eivindjahren]

The current test setup is hard to work with for two reasons:

• Its difficult to know where a test is placed, and where to place your test
• Some tests take a long time to run
• Some tests create GUI elements, which is hard to run locally while working
• Some tests care too much about implementation detail

There are several markers that are used to enable/disable such tests

• quick_only
• slow
• requires_window_manager
• integration_test

The directory layout of tests is difficult for several reasons, first its the top level directories ert_tests and libres_tests which has no semantic meaning anymore. Following is a suggested layout, and guidelines for what test goes where should be put in appropriate README.md files inside of test/:

tests/
│   README.md
│   conftest.py    
│
└───unit_tests/
│   └───test_{super_module}/
│       │   __init__.py
│       │   test_{sub_module1}.py
│       │   test_{sub_module2}.py
│       │   ...
│   
└───gui_tests/
    │   __init__.py
    │   test_{gui_feature1}.py
    │   test_{gui_feature2}.py
│   │   ...
└───integration_tests/
    │   __init__.py
    │   test_{system1}.py
    │   test_{system2}.py
└───performance_tests/
    │   __init__.py
    │   test_{system1}.py
    │   test_{system2}.py

Guiding ideas:

• Unit tests should (as much as possible) test the public interface of a module (which could be .py a file or a directory containing an __init__.py) in isolation.
• Unit tests should run fast (we want more of them), sub 1 second under normal conditions per case (A parametrized test has many cases) is a hard requirement.
• Only gui tests are allowed to create windows.
• Integration tests should create a reasonable amount of coverage for the time it takes to run the test (if it takes a lot of time it should fail for many potential bugs).

[frode-aarstad]

What about the c-tests? How should we structure code that is common for several tests (fixtures etc) ?

[eivindjahren]

What about the c-tests? How should we structure code that is common for several tests (fixtures etc) ?

I think perhaps we can treat ert._clib as a super module, and that some tests are written in c and therefore go under tests/unit_tests/test_clib/. A quick prototype by @zohar showed that it is doable with CMake.