practices
practices copied to clipboard
canonical-web-and-design
Document standard test policy
This document should answer those questions:
- What should we test (complex functions?, long functions?) ?
- Unit tests vs Integrations tests, e2e tests (:+1::+1::+1::+1::+1: )
- How should we test (libraries)?
So I'll have a crack at the "what to test" question.
-
As a basis, we should only test the behaviour of applications/libraries/modules, not the internals
Technically, this often means testing interfaces rather than units/functions, but of course there's fuzziness there as to what "interfaces" means, or which interfaces to test. I think that is a nuanced decision and depends on the project. Here are some suggestions:
- For a website, test HTTP endpoints or view functions. Try not to go deeper into testing the constituent parts of an individual view (ideally significant functionality would be extracted into its own module anyway)
- For modules, test the interfaces to the module. E.g. any public classes or functions. Don't test internal functions.
-
Unit tests for specific pieces of functionality may be written when a particular unit is known to be unusually complex or problematic (although rewriting it to be simpler would be ideal)
The reason for this high-level approach is that I want to avoid ending up with codebases that people don't dare to change for fear of having to rewrite hundreds of tests. This can be a serious damper on innovation and flexibility.
For Python tests, the two libraries I'd like to consider are unittest and pytest.
I was discussing this with @WillMoggridge yesterday - we think PyTest is basically built on top of unittest, and adds more sugar. My concern with PyTest is that it's harder to run naively - you can't simply throw in an ipdb debugger to debug a test, as all tests need to be run through the pytest CLI tool, whereas unittest scripts can simply be run like any other Python script.
If someone can show me how to write PyTest scripts which can be naively run with python test.py and easily debugged then I'm totally happy to standardise around PyTest.
We should also consider that Django has its own testing framework. Should we be using that for Django projects? Or does it make just as much sense to write tests for Django in whatever our standard library is instead?
For reference:
- documentation-builder uses PyTest
- snapcraft.io uses unittest
-
assets.ubuntu.com and partners.ubuntu.com use
django.test
I'm also open to suggestions of other frameworks to use.
Almost a year down the line, do @WillMoggridge, @tbille or @jpmartinspt have any further opinions about Python testing frameworks?
Necro bump :skull:
I don't know if python testing libs are still being evaluated, but testtools is excellent. It's built on unittest and allows you to compose custom matchers that make for expressive and readable tests.
See Matchers, better than assertion methods
Snap delta service, and all of the snap store microservices use testtools.
Yeah I had a look at testtools a while back - someone in snap store mentioned it to me. It's got useful extensions, but it's so much less well known or used than the standard unittest, I think maybe we should recommend unittest as the standard, but mention testtools as an optional extension if you need the functionality.
I'll try to write this doc soon.
it's so much less well known or used than the standard unittest, I think maybe we should recommend unittest as the standard, but mention testtools as an optional extension if you need the functionality.
It's actually an extension of unittest, rather than a distinct framework like pytest. Anyone familiar with unittest will find testtools familiar. Starting with unittest sounds sensible though.
I agree with starting with unittest. For django we should also mention their extension of the unittest.TestCase as it provides useful features for testing django projects.
