benchexec
benchexec copied to clipboard
sosy-lab
Update test framework
Currently the tests are executed with nose, but this framework is dead. We should switch to a new framework, for example nose2 or py.test.
It seems py.test has quite useful support for fixtures and parameterized tests, not sure whether nose2 also supports this.
Furthermore, setup.py test is deprecated and will be removed in the future. We might just tell people to use the command of the test framework that we decide on, or use tox.
Problems I found while investigating:
- We have tests directly in packages (in
__init__.py), but nose2 and other testing frameworks do not support this. - We have test modules with names that are not globally unique (e.g.,
test_integration), this also seems problematic for many testing frameworks. - We have tests below
contrib/, butcontrib/is not a proper package (and shouldn't be).
Hi, can you explain what we need to do to fix the issue, I am a beginner so I am having trouble understanding.
As a first step, an appropriate framework needs to be chosen. This should be done systematically based on criteria like ease of use, how common the framework is, whether it is well maintained etc. Next, the framework needs to be integrated in BenchExec's project structure and the existing tests need to adjusted where necessary. This might also include restructuring the tests and in general improving their state because as mentioned above, the current test infrastructure might be problematic for some test frameworks.
I have created a new discussion related to this topic as it can easy to find for everyone instead of under an issue
Yes, please let's keep this issue focused on actual discussion of the migration to a new test framework. Any more general discussions, including personal introductions, GSoC projects, etc. should be done in the GitHub Discussions.
Hi, Can you explain what we need to fix the issue, I am a beginner so I have some problems to understanding.
There are some concrete problems listed above and there is also a discussion about this in https://github.com/sosy-lab/benchexec/discussions/903. If you have any further concrete questions, please ask them. Questions about the concrete issue should be posted here, more general questions in the discussion.