Test selection does not work

[tuukkamustonen]

I started up a test project, created one single app there with a testcase extending from UnitTestCase.

I can successfully run tests through nosetests. However, picking specific testcase type's usting sanetestselectionplugin does not seem to work. Running:

PYTHONPATH=.:.. DJANGO_SETTINGS_MODULE=settings nosetests --with-django --with-sanetestselectionplugin -u

finds 0 tests despite the existing UnitTestCase in the project.

If I change my testcase to extend DatabaseTestCase and try to run only database tests, nose still finds 0 tests.

However, if I extend my testcase from either django.test.testcases.TestCase or simply unittest.case.TestCase, nose finds the tests despite what selectors I try to pass to with through --with-sanetestselectionplugin

[Almad]

Strange, I'll dig into that.

Would You mind uploading some example code so I can verify on it?

[tuukkamustonen]

Try this:

mkvirtualenv --no-site-packages sanetest pip install django djangosanetesting nose yolk yolk -l # to see the installed versions django-admin.py startproject sanetest cd sanetest/ nano settings.py # set database ENGINE to django.backends.db.sqlite3 django-admin.py startapp runnertest PYTHONPATH=. DJANGO_SETTINGS_MODULE=settings nosetests --with-django # runs the test automatically inserted by django-admin nano runnertest/tests.py # set the SimpleTest class extend from djangosanetesting.cases.UnitTestCase instead PYTHONPATH=. DJANGO_SETTINGS_MODULE=settings nosetests --with-django --with-sanetestselectionplugin -u # finds 0 tests?

[Almad]

Hi,

I will verify, but if I recall correctly, this is because default test is in tests module, which is not in test* directory and thus skippped by nose discovery mechanism.

I should probably explain this in docs in more detail and probably bundle something like --djangodiscovery plugin, which would look for applications in installed_apps and their test module.

[tuukkamustonen]

Nose does find the tests inside tests.py. However, when overriding testcase selection by --with-sanetestselectionplugin, the test discovery gets broken. To me, it seems that the plugin breaks discovery, not nose. Please feel free to educate me if I'm all wrong :)

[Almad]

Yeah, this looks like a bug :]

Just I forgot to ask -- can You please try it with development version? This might be fixed with #30...if yes, I'll release a fix for 0.5 branch sometimes next week.

[Almad]

Now back on my computer and normal internet so I can introspect :)

Well, the problem is that with nose discovery, nose searches for objects matching NOSE_TESTMATCH or tests inheriting from unittest.TestCase.

Thus, if you will run

PYTHONPATH=. DJANGO_SETTINGS_MODULE=settings ../bin/nosetests --with-django runnertest/tests.py -m '^Simple*|^run*'

it will work as advertised.

As said, to "fix" this bug, I'll probably introduce a plugin that will look for SaneTestCase subclasses as an alternative way to specify testcase.

[tuukkamustonen]

Thanks for looking into it. However:

As said, to "fix" this bug, I'll probably introduce a plugin that will look for SaneTestCase subclasses as an alternative way to specify testcase

Isn't --with-sanetestselectionplugin supposed to do exactly that? If not, what does it do then?

[Almad]

Nope -- it is supposed to let you specify only one type of test (i.e. unittests) and ignore other types (http test et al).

Thanks for the feedback -- alternative discovery should probably be part of this plugin and I'll documentation should be more verbatim about that.

[tuukkamustonen]

Sorry to constantly get back on this but I did try to select only unittests by providing -u option. Check the original report. Am I really missing something now?

[Almad]

Nothing to be sorry about, this is what I get for blurry documentation ;)

"unittest" is in this case not stdlib's unittest, but UnitTestCase from DST, i.e. plugin is looking for discovered classes and looking for "test_type" attribute being equal to "unit".

However, it is not overwriting standard discovery, so if testcase is not matching NOSE_TESTMATCH, it is not discovered and fed to plugin...which is confusing and should be fixed, as said :)

(also, documentation should be more clear about "unittest(2).TestCase" and "UnitTestCase" distinction.

[tuukkamustonen]

Aah so nose doesn't find the test anymore because I changed it to inherit from something else than unittest.TestaCase (and also because the file is not in discovery path but it never was). Gotcha.

Closing this ticket (though I wonder why SaneTestCases do not extend from unittest.TestCase, that's actually something that caught my eye in the first place)... anyway, thanks for clarfying things up!

[Almad]

If you don't mind, I'd leave it open until doc/discovery fix ;)

Some nose features are disabled with unittest.TestCase, that's why...

[ncoghlan]

I'm pretty sure I just got caught by this issue. I'm in the process of setting up the test infrastructure for a new project, so I only have a single test that checks the index page displays without error. Switched over to sane-testing via TEST_RUNNER option in settings.py and the test ran fine, but as soon as I switched the test itself to inherit from djangosanetesting.cases.HttpTestCase instead of django.test.TestCase, the test execution claimed I no longer had any tests defined.

Is there a workaround for this that allows "./manage.py test" to discover the tests correctly?

[Almad]

Moving this up on my TODO list.

@ncoghlan: I am afraid only by editing command :/

I am using something along the lines of https://github.com/Almad/django-sane-testing/blob/master/pavement.py#L70 everywhere, but I'll fix it for You ;)

[ncoghlan]

Thanks!

I was mainly happy to discover that there was a reason for the strange behaviour I was seeing - now that I know about it, I can make my test suite work with it (most likely by adding a shell script or Makefile command that invokes nosetests directly, along the lines of the workaround you posted above)

[ncoghlan]

After reading up a bit more on how nose does its pattern matching, I found a really simple fix: I renamed my test case from "IndexTestCase" to "Index_TestCase". The underscore lets the default regex find the "Test" in the name, while the camelcase naming confuses it.

So long as I continue to follow that convention, the automatic test discovery will work correctly.

[Almad]

Thanks for this hack, I'll put it into docs.