geo-deep-learning
geo-deep-learning copied to clipboard
NRCan
Automated testing and code coverage
Hi,
I did not find any issues directly related to automated testing or code coverage. Are the automated tests kept locally? Do you have a testing strategy for the project? Is there any interest or need to develop a automated testing infrastructure with your continuous integration?
Hi,
We do not yet have a "full" test strategy. We just make sure nothing breaks after we commit code with Travis-CI.
Since our tool is commandline based, I would be inclined to at least do Doctests. We are still unsure how pervasive tests should be and what tools to use. Suggestions ?
Hi Yves,
Sorry for the delay in my response. I wanted to take the time to formulate a complete answer regarding my suggestions.
There are different test strategies that can be adopted to fit your needs. The best thing that can be done is to start having the conversation about how to test your projects and bring your test coverage up.
Before explaining some options, I just want to rapidly review why testing is necessary for any software project.
Test automation enables us to answer the following questions:
- How can you consider if your code works the way it was intended to?
- How can you know you have not introduced a bug with a recent change?
- How can you efficiently test if you have not introduced a bug every time a change is made if you do not have automated tests?
The long-term effects of having tested code (ideally with maximal coverage) include
Lowering the cost of software on its entire life cycle. This is especially evident when it comes to the maintenance of software over time.
Improving developer productivity by enabling them to practice aggressive refactoring.
Serving as documentation for developers. The intent of our code if ubiquitously presented in the tests.
Improving the design of our code. For code to be easy to test, it must be well-designed.
Project Adoption. Software that is tested is more likely to be adopted, especially in situations where system failures are dramatic. Tested software is considered high quality, less prone to breaking.
The first step is defining acceptance tests from your project requirements. Acceptance tests are tests defined by programmers and stakeholders that define when a requirement is done. Acceptance tests should be automated simply because it reduces the cost of testing compared to manually inspecting if every acceptance test is still respected when refactoring.
There are then many other types of tests that should be included in your automated test suite. A typical pattern presented is a pyramidal format where (a) Unit Tests are at the bottom, followed by (b) Component Tests, (c) Integration Tests, (d) System Tests and Exploratory Tests.
(a) Unit tests are the tests written by programmers the can serve as documentation for future developers on the same project. They specify the system at the lowest level. In a TDD approach, these tests specify what the developer is about to write. A coverage as near as 100% should be aimed for.
(b) Component tests are the acceptance tests defined previously. They are the tests that cover the business rules. Usually carried out by passing data to a component and evaluating its output.
(c) Integration tests test the interaction between different components. They test the assembly of different parts together.
(d) System Tests are tests that are executed against the entire system. They do not test business rules directly, they can be considered as the ultimate integration tests. Usually carried out by technical leads and system architects.
(e) Manual Exploratory Tests are manual tests done by human with a creative and new look on the system. They intent is to explore the system for unexpected behaviour that was not caught by the scripted tests. Can sometimes be carried out by specialist or by an entire team for a day. The goal is not coverage but the ease of use of the system.
More specifically for your project, given the state in which it is already advanced, I would seek to define acceptance tests of the different components. Once those tests are defined, increasing your integration tests of the components and your unit tests would be the next logical step. The integration tests could be carried out by a specific developer/architect while the unit tests should be the responsibility of the entire team.
It can seem like a lot of work for a system that is already working, but it will be at lot less work when the system breaks. Software failure is a question of when not if.
Since most of your development seems to be in Python, the most adopted testing solution is pytest. Pytest can easily be integrated in your continuous integration with Jenkins it would automatically execute your test suite on commits.
Coverage is in interesting tool to get an idea of the test coverage of a project. It is not a silver bullet, but it can provide some metric of how the evolution of your test coverage is improving over time.
Hypothesis (property based testing) is also a personal favorite when it comes to generating a tone of different data for your tests.
If you have any other questions or would like to discuss about some specifics regarding your test strategy do not hesitate to reach out.
One I like very much is "Single assert statement(assert a == b) over 40 different assert methods(self.assertEqual, self.assertIsInstance, self.assertDictEqual)"
