mailinabox
mailinabox copied to clipboard
mail-in-a-box
starting a test suite (#777)
This is how I imagine the miab test suite to look like. For now it tests ssh login, https redirect for all services and the core mail functionality (smtp, pop3, imap). If this fits miab's vision of a test suite, I can add some more tests. Looking forward for your comments and suggestions
The tests are using the excellent pytest tool and require a working vagrant box. Individual tests are just functions with a name starting with "test_" and at least one assert statement. pytest auto-discovers those tests and runs them. So it's fairly easy to add additional tests and automating the testing (for example with travis) is possible.
The current tests are mainly adapted from the existing tests and from sovereign.
for the note: testing with travis won't work as we've got to spawn a full VM with many services. Memory requirements with travis are ridiculously low. The best alternative I can think of is gitlab-ci, as you can host your own runners (at home or anywhere else), thus controlling how much memory and what it can do.
Great work! I was busy (slowly and nowhere near finished) with writing unit tests on the management interface using flask. I should be able to integrate that in here.
Thx for the feedback!
@guyzmo Yes, travis could be problematic. It would be nice if we can avoid hosting our own CI tool and find a hosted one that works with vagrant. I still need to look into that.
@yodax I don't see yet why you would need flask for the tests, but I am sure there is a good reason for that. For the management interface I am currently just testing that the https redirect works and if it's accessible. So there are definitely many things untested.
For now, I am working on some tests for the caldav interface.
I don't see yet why you would need flask for the tests, but I am sure there is a good reason for that.
I'm working on testing fault situations; like UFW not being installed crashing the status checks.
Just mentioned it because I'm happy that it should be easy to integrate it into your work.
@wsteitz I'm following along with the new commits - thanks for working on this!!
Thanks @wsteitz for your work on the test suite. I wanted to look into helping out with tests as well so I took your testsuite branch, rebased it onto current master and fixed a few issues that came up. Feel free to pull them from yeah/testsuite.
@JoshData would it be possible to merge this PR afterwards? It would make it much easier for others to get involved and submit PRs with more tests. (Happy to work on a few.)
For sure. Once the next release goes out my main focus is on testing and getting to Ubuntu 18.04.
Feel free to open any issues and tag them with „help needed“ as you see fit. Happy to help as best as I can. Getting to full coverage is a lot of work. I think you will have the best idea on how to prioritize, so any guidance on where to begin is much appreciated.
for the note: testing with travis won't work as we've got to spawn a full VM with many services. Memory requirements with travis are ridiculously low.
Travis CI provides a full VM with 7.5 GB of memory. I set it up and was able to successfully install Mail-in-a-Box. See the log here. I can submit a pull request, if anyone is interested.
