nose icon indicating copy to clipboard operation
nose copied to clipboard

Usability issue with re-run only failed tests (--failed flag implementation in testid.py )

Open p00j4 opened this issue 7 years ago • 0 comments

this issue may be debatable that its a bug or feature hence here is the example why it should be considered a bug. Actual : as per document it works perfectly fine with normal whole run which fails at-least 1 test-> failed only run -> fix issue -> failed only run -> failed only run sequence

but if run like this, it runs a lot of previous failed tests normal whole run which fails at-least 1 test -> fix issue -> normal whole run again (which passes those tests) -> failed only run (this still runs the failed once )

Expected if i'm not deleting .noseids periodically, i'm ending up with a lot of tests in failed category because most of the times, I use whole run and when finally I run "failed only" run, even the very older failed once and for obvious reasons, consumes more time.

Example

  1. simple 3 tests test.py
import unittest
class Test1(unittest.TestCase):
    def test_no1(self):
        assert 1 == 1
    def test_no2(self):
        assert 1 == 2
    def test_no3(self):
        assert 1 == 3

run with : nosetests test.py --with-id -vv
which shows "Ran 3 tests (2 failures)" Observe: .noseids file stores the failed detail, example like : self.failed = ['2','3'] (testids given in same order let's say 1, 2, 3)

  1. now fix one of the issue
import unittest
class Test1(unittest.TestCase):

    def test_no1(self):
        assert 1 == 1

    def test_no2(self):
        assert 2 == 2

   def test_no3(self):
        assert 1 == 3

and run again whole which shows "Ran 3 tests run(1 failure)" nosetests test.py --with-id -vv

now run failed only tests which shows - "Ran 2 tests run(1 failure)"

Observe: even if there is only 1 test failure in last state, still on runing failed only, it executes 2 tests instead of 1 only.
Possible Root Cause: because .noseids file doesn't get updated, it should have been to "failed:'3' " ( because the self.failed still points to = ['2','3'] , as per code looks like it gets updated only on run with "--failed" )

now, "failed only" run will execute this both, ideally should have executed only '3'

the bigger problem: Let say, 1000s of test cases and 50 of them failed but corrected 49 and kept running like "normal whole run", and when ran with "failed" still all 50 will be executed and imagine using nose of functional testing and each test takes significant amount of time.

p00j4 avatar Jan 31 '17 15:01 p00j4