Circumvent freezegun in favor of a real clock where possible

[jombooth]

Hi,

First off, thank you for this wonderful tool! At my work we've been using it to get CircleCI's Test Summary feature to display information about our test runs, and it's been awesome.

I noticed however that for certain test suites where we use Freezegun, the test runner reports a test runtime of 0.000s. I saw in this comment that some prior efforts have been made to circumvent freezegun in this project; unfortunately, this approach did not seem to work (at least, not in our case).

Following the pseudocode in this pep, I implemented a function that's typically equivalent to authentic time.time(), but unlike that function, is not mocked out by freezegun. This is now in xmlrunner.time, and used everywhere time() was previously called in this project. Where the clock_gettime call it depends on isn't available (mostly, non-Unix systems), or where there is no CLOCK_REALTIME available on the system, the new function will fall back to (possibly mocked) time.time().

I also verified in unit tests of the builder and the runner that the resulting timestamps + elapsed time evaluations are accurate even for a test during which time is frozen with freezegun - this necessitated adding freezegun to the test env in tox.ini.

tox -e pytest works fine with the new tests; I apparently didn't have some of the interpreters running the tests with tox requires, so that command failed for me locally, but I expect there won't be issues on a properly prepared system.

[coveralls]

Coverage decreased (-0.2%) to 99.431% when pulling abf0ec97697bc75853f659190cec18ff531a3e14 on jombooth:jo-circumvent-freezegun into 7d0a0e7e946f5c44593f53dcc3cd24d641db93de on xmlrunner:master.

[codecov-commenter]

Codecov Report

Merging #228 into master will decrease coverage by 0.17%. The diff coverage is 94.00%.

@@            Coverage Diff             @@
##           master     #228      +/-   ##
==========================================
- Coverage   99.61%   99.43%   -0.18%     
==========================================
  Files          15       16       +1     
  Lines        1566     1607      +41     
==========================================
+ Hits         1560     1598      +38     
- Misses          6        9       +3     

Impacted Files	Coverage Δ
xmlrunner/time.py	62.50% <62.50%> (ø)
tests/builder_test.py	100.00% <100.00%> (ø)
tests/testsuite.py	100.00% <100.00%> (ø)
xmlrunner/builder.py	100.00% <100.00%> (ø)
xmlrunner/result.py	98.61% <100.00%> (ø)
xmlrunner/runner.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 7d0a0e7...abf0ec9. Read the comment docs.

[jombooth]

Hi, just bumping this! Re: the coverage checks - I'm happy to follow up with more testing but just wanted to confirm that this change is going in a good direction so far.

[dnozay]

@jombooth - can you please complete the 5 whys exercise for this PR? e.g. why do you need to use a a real clock?

[jombooth]

Thanks for your reply! This is maybe the second time I've ever done the "five whys" exercise, so I'll give it a shot but no guarantees this'll be the best representation of the technique:

• Why [do I want to use a real clock]? Because currently, when I run the test suite in the large project where I'm using xmlrunner, all the tests where we use Freezegun report completing in 0.000s, both in the XML reports and on the command line. I want to get real numbers here.
• Why? Because I'd like to understand how long my tests are actually taking, and am pretty bought into freezegun (a very popular time-mocking package - also like I mentioned, there was a previous attempt at this (https://github.com/xmlrunner/unittest-xml-reporting/pull/185)).
• Why? Because data about slow tests can make it easier to pinpoint suites that could be optimized or removed. Data on especially fast tests can also reveal cases where the test isn't efficacious - ie, where the function called is unintentionally returning immediately or something
• Why? Because unefficacious tests provide anti-value, falsely reassuring programmers that the code is being tested, and because slow tests make for slow CI
• Why? Because I want my projects' tests to inspire only warranted confidence that the code is correct, and because quick CI makes for quick iteration, which makes for quick development

[dnozay]

@jombooth , maybe this could also be the quick fix for you:

import freezegun
freezegun.configure(extend_ignore_list=['xmlrunner'])

https://github.com/spulec/freezegun#ignore-packages