load tests: pass argument to pytest to support different modes

[tobiasschweizer]

I think it would make sense if we could pass arguments to pytest in order to support different modes. I would like to have a travis and a local mode for the load tests.

In CMakeLists.txt, I would like to support the following logic:

option(MODE_TRAVIS "Run load tests with reduced load" ON)

if(MODE_TRAVIS)
    # will be run when `make test` is executed
    add_test(NAME all_e2e_tests
        WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test/e2e
        COMMAND pytest --mode="travis")
else()
    # will be run when `make test` is executed
    add_test(NAME all_e2e_tests
            WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test/e2e
            COMMAND pytest --mode="local")
endif()

Then in test_02_test_server.py the argument mode would have to be checked. I guess this could work like documented here: https://docs.pytest.org/en/latest/parametrize.html#basic-pytest-generate-tests-example.

[subotic]

Why not try to make the test itself more adaptive, i.e. count the number of CPUs and then use this for the load? This way, this test will run everywhere as intended and not only on your machine and Travis.

[subotic]

Also, keep in mind, that long-running tests should preferably only run on Travis. Better to have a flag in CMakeLists.txt so that we can skip long-running tests locally. Just a thought.

[tobiasschweizer]

Why not try to make the test itself more adaptive, i.e. count the number of CPUs and then use this for the load? This way, this test will run everywhere as intended and not only on your machine and Travis.

That would be great! For the load tests, the configuration must be adapted to the system they are running on. Otherwise they cannot possibly pass.

Do you have a practical solution at hand?

Also, keep in mind, that long-running tests should preferably only run on Travis. Better to have a flag in CMakeLists.txt so that we can skip long-running tests locally. Just a thought.

Actually I thought of a argument you could pass to make test.

[subotic]

Maybe we could use something like this: https://stackoverflow.com/a/1006301

We know how the test is configured for Travis-CI. There we have only two CPU cores available. Basically, we could use this as the baseline and multiply by the CPU count.