django-sane-testing
django-sane-testing copied to clipboard
Test selection does not work
I started up a test project, created one single app there with a testcase extending from UnitTestCase.
I can successfully run tests through nosetests. However, picking specific testcase type's usting sanetestselectionplugin does not seem to work. Running:
PYTHONPATH=.:.. DJANGO_SETTINGS_MODULE=settings nosetests --with-django --with-sanetestselectionplugin -u
finds 0 tests despite the existing UnitTestCase in the project.
If I change my testcase to extend DatabaseTestCase and try to run only database tests, nose still finds 0 tests.
However, if I extend my testcase from either django.test.testcases.TestCase or simply unittest.case.TestCase, nose finds the tests despite what selectors I try to pass to with through --with-sanetestselectionplugin
Strange, I'll dig into that.
Would You mind uploading some example code so I can verify on it?
Try this:
mkvirtualenv --no-site-packages sanetest pip install django djangosanetesting nose yolk yolk -l # to see the installed versions django-admin.py startproject sanetest cd sanetest/ nano settings.py # set database ENGINE to django.backends.db.sqlite3 django-admin.py startapp runnertest PYTHONPATH=. DJANGO_SETTINGS_MODULE=settings nosetests --with-django # runs the test automatically inserted by django-admin nano runnertest/tests.py # set the SimpleTest class extend from djangosanetesting.cases.UnitTestCase instead PYTHONPATH=. DJANGO_SETTINGS_MODULE=settings nosetests --with-django --with-sanetestselectionplugin -u # finds 0 tests?
Hi,
I will verify, but if I recall correctly, this is because default test is in tests module, which is not in test* directory and thus skippped by nose discovery mechanism.
I should probably explain this in docs in more detail and probably bundle something like --djangodiscovery plugin, which would look for applications in installed_apps and their test module.
Nose does find the tests inside tests.py. However, when overriding testcase selection by --with-sanetestselectionplugin, the test discovery gets broken. To me, it seems that the plugin breaks discovery, not nose. Please feel free to educate me if I'm all wrong :)
Yeah, this looks like a bug :]
Just I forgot to ask -- can You please try it with development version? This might be fixed with #30...if yes, I'll release a fix for 0.5 branch sometimes next week.
Now back on my computer and normal internet so I can introspect :)
Well, the problem is that with nose discovery, nose searches for objects matching NOSE_TESTMATCH or tests inheriting from unittest.TestCase.
Thus, if you will run
PYTHONPATH=. DJANGO_SETTINGS_MODULE=settings ../bin/nosetests --with-django runnertest/tests.py -m '^Simple*|^run*'
it will work as advertised.
As said, to "fix" this bug, I'll probably introduce a plugin that will look for SaneTestCase subclasses as an alternative way to specify testcase.
Thanks for looking into it. However:
As said, to "fix" this bug, I'll probably introduce a plugin that will look for SaneTestCase subclasses as an alternative way to specify testcase
Isn't --with-sanetestselectionplugin
supposed to do exactly that? If not, what does it do then?
Nope -- it is supposed to let you specify only one type of test (i.e. unittests) and ignore other types (http test et al).
Thanks for the feedback -- alternative discovery should probably be part of this plugin and I'll documentation should be more verbatim about that.
Sorry to constantly get back on this but I did try to select only unittests by providing -u
option. Check the original report. Am I really missing something now?
Nothing to be sorry about, this is what I get for blurry documentation ;)
"unittest" is in this case not stdlib's unittest, but UnitTestCase from DST, i.e. plugin is looking for discovered classes and looking for "test_type" attribute being equal to "unit".
However, it is not overwriting standard discovery, so if testcase is not matching NOSE_TESTMATCH, it is not discovered and fed to plugin...which is confusing and should be fixed, as said :)
(also, documentation should be more clear about "unittest(2).TestCase" and "UnitTestCase" distinction.
Aah so nose doesn't find the test anymore because I changed it to inherit from something else than unittest.TestaCase
(and also because the file is not in discovery path but it never was). Gotcha.
Closing this ticket (though I wonder why SaneTestCases do not extend from unittest.TestCase
, that's actually something that caught my eye in the first place)... anyway, thanks for clarfying things up!
If you don't mind, I'd leave it open until doc/discovery fix ;)
Some nose features are disabled with unittest.TestCase
, that's why...
I'm pretty sure I just got caught by this issue. I'm in the process of setting up the test infrastructure for a new project, so I only have a single test that checks the index page displays without error. Switched over to sane-testing via TEST_RUNNER option in settings.py and the test ran fine, but as soon as I switched the test itself to inherit from djangosanetesting.cases.HttpTestCase instead of django.test.TestCase, the test execution claimed I no longer had any tests defined.
Is there a workaround for this that allows "./manage.py test" to discover the tests correctly?
Moving this up on my TODO list.
@ncoghlan: I am afraid only by editing command :/
I am using something along the lines of https://github.com/Almad/django-sane-testing/blob/master/pavement.py#L70 everywhere, but I'll fix it for You ;)
Thanks!
I was mainly happy to discover that there was a reason for the strange behaviour I was seeing - now that I know about it, I can make my test suite work with it (most likely by adding a shell script or Makefile command that invokes nosetests directly, along the lines of the workaround you posted above)
After reading up a bit more on how nose does its pattern matching, I found a really simple fix: I renamed my test case from "IndexTestCase" to "Index_TestCase". The underscore lets the default regex find the "Test" in the name, while the camelcase naming confuses it.
So long as I continue to follow that convention, the automatic test discovery will work correctly.
Thanks for this hack, I'll put it into docs.