unittest API for detecting test failure in cleanup/teardown

[bitdancer]

BPO	24249
Nosy	@rbtcollins, @ezio-melotti, @bitdancer, @voidspace, @vadmium

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-05-20.21:01:55.280>
labels = ['easy', 'type-feature']
title = 'unittest API for detecting test failure in cleanup/teardown'
updated_at = <Date 2020-01-29.03:22:47.048>
user = 'https://github.com/bitdancer'

bugs.python.org fields:

activity = <Date 2020-01-29.03:22:47.048>
actor = 'josephgordon'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2015-05-20.21:01:55.280>
creator = 'r.david.murray'
dependencies = []
files = []
hgrepos = []
issue_num = 24249
keywords = ['easy']
message_count = 5.0
messages = ['243694', '243703', '243704', '243720', '246703']
nosy_count = 6.0
nosy_names = ['rbcollins', 'ezio.melotti', 'r.david.murray', 'michael.foord', 'martin.panter', 'azsorkin']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24249'
versions = ['Python 3.6']

Linked PRs

• gh-121709

[bitdancer]

I have a situation where it would be really helpful to know in my cleanup routine whether or not the test failed. The situation is this: I'm running a command in a subprocess, and sometimes it writes output to stderr that I normally want to ignore. But if the test fails, I'd like my cleanup routine (that shuts down the test subprocess) to print out the stderr, because usually the information as to what went wrong is in stderr.

I figure out this hack (for python3.4):

    for what, result in self._outcome.errors:
        if what._testMethodName == self._testMethodName and result:
            print(self.p.stderr.read().decode())

but obviously that uses non-public APIs.

[vadmium]

Can’t you use the -b or --buffer command-line option <https://docs.python.org/dev/library/unittest.html#cmdoption-unittest-b>, or equivalent option to unittest.main() or whatever?

The standard output and standard error streams are buffered during the test run. Output during a passing test is discarded. Output is echoed normally on test fail or error and is added to the failure messages.

[vadmium]

Ah sorry I didn’t see the keyword “subprocess”, so --buffer won’t work! But maybe you can dump the subprocess’s stderr to sys.stderr?

[bitdancer]

Having to specify a command line option to get a normally clean test run just doesn't feel like a good API. If there is a way to set that on the TestCase, that would be OK.

There are other reasons to want the API I suggest, though...there is at least one StackOverflow question asking about how to do it, which is where I got the clue for implementing my hack.

[rbtcollins]

Setting some basic design parameters here.

Its ok for a test to know if it itself has decided on some status. See e.g. testtools.expectThat for a related design thing.

Making some official API within the test itself so code after the failure can take appropriate actions seems pretty safe to me, as long as its a one-way switch - no backsies.

I think inspecting the reporter would be overly limiting on the reporter implementation, and we shouldn't do that.

Further, I think limiting the possible status's that we track to the minimum would be good here, even though its inconsistent with the current overly rich separation unittest offers.

That is, some self.status field which might even be an enum (if enum34 supports Python 2.6 for backports [as unittest is currently backported to 2.6 and up].

unset success unexpected success skipped failed expected failure

[stinovlas]

I'll try to tackle this on EuroPython 2024 sprint.