Improve code coverage

[Technologicat]

trafficstars

Coverage is currently at 85%. At least some of the final 15% would be nice to add.

A few percent could be false positives from the use of block macros in the tests for those block macros, or trivial error handling code that's not triggered by the tests, but there can be some exotic cases that are genuinely missing.

There is now a a CI setup that runs the test suite on Pythons 3.4, 3.5, 3.6, 3.7 and PyPy3, as well as runs coverage.py and uploads the coverage report to codecov.io.

Significant improvements in code coverage are slow work, and may have to wait until 0.14.4 or later.

See #5 for history.

[Technologicat]

Doing some of this already in 0.14.3.

It's been useful, a number of small bugs have been discovered and fixed.

[Technologicat]

Currently at 89%.

The remaining missing (i.e. non-covered) lines fall into a few categories:

• False negatives, mostly related to macro expansion of block macros (https://github.com/nedbat/coveragepy/issues/1004), plus some module docstrings.
• Python version specific code, not run under CPython 3.6, where coverage is currently measured.
  • Code specific to 3.4 or 3.5 has been tagged # pragma: no cover. Code for 3.7 and 3.8 needs to be re-checked (grep: Python 3.7, Python 3.8) once we bump the minimum Python version.
  • Code specific to PyPy3 has at least mostly been tagged # pragma: no cover, but this could be revisited. Have to first check PyPy's plans for 3.7+ support, to know how relevant it will likely be in the future...
• Code whose tests are currently disabled due to incompatibility with Python 3.4 syntax (which would crash the whole test suite while compiling the relevant module, if running under 3.4).
  • E.g. there is a case in namedlambda, in unpythonic.syntax.lambdatools, which would be triggered by the tests, but the syntax used the tests (to trigger that code) requires the * and ** unpacking generalizations added in 3.5 and 3.6.
  • There should be a warn[] in the corresponding tests, so it can be fixed later when we bump the minimum Python version.
• Code whose tests are currently disabled due to https://github.com/azazel75/macropy/issues/26. Basically anything dealing with bytestrings or complex numbers. Affects especially unpythonic.net.* and unpythonic.typecheck.
• Test framework (unpythonic.test.fixtures, unpythonic.syntax.testingtools), particularly code handling test failures and errors. Cannot use the test framework to test itself.
• Network components (unpythonic.net). Need to think of something here. Not really robust without automated tests, but it's the kind of interactive stuff that's hard to test compactly.
• Tricky (often corner) cases:
  • Two in the continuations macro in unpythonic.syntax.tailtools. Still have to figure out what exactly I was thinking when I coded that.
  • The multithreading paranoia in unpythonic.singleton and unpythonic.symbol. The code seems correct to me. I can't think of a reason that the final check shouldn't be there, but haven't found a way to cause it to trigger, either.
  • One in unpythonic.arity._kwargs.
  • A __setstate__ in unpythonic.test.test_singleton.

Notable exclusions:

• Any code that emits a macro expansion error (assert False, "...message..." in macro code).
• Import failure handlers for optional dependencies. We require those dependencies for development in requirements.txt.
• Failure code paths in tests.
• The "if main" idiom in the individual test modules. (Allows them to run individually as main programs through the macropy3 wrapper, but runtests.py imports them as modules.)
• Most of the custom __repr__.
• Anything that catches UnknownArity.
• Quote blocks in macros. (Quoted code is not even supposed to run at that point, obviously.)
• Some paranoia code.
  • In mogrify: is there any realistic MutableSequence that has no clear(), or MutableSet that has no update()? The API for those ABCs allows that, so it's checked and worked around if it happens. But e.g. the builtin list has clear() and set has update(), so it's unlikely any client code will ever run into that case.
  • In isoftype: the implementation failure guards.
• A __del__ in unpythonic.dynassign. It certainly runs, but isn't reported as covered for some reason. So it's excluded to stop coverage.py from complaining about it.
• A MacroPy 1.1.0b2 fallback in unpythonic.syntax.lambdatools.
• Some backward compatibility aliases, e.g. escape (now throw) and setescape (now catch).

[Technologicat]

Now at 91%.

• Beside just improving test coverage, we should aim at high coverage (as high as reasonable) in each module's own unit tests.
  • Currently (esp. in low-level utilities) there's some code that's only exercised by tests of other modules. Hence a module, when tested individually, may have lower test coverage than it has when the full test suite runs. If that module breaks, this may make bugs harder to find than necessary.
  • In some cases, such as lazyutil, that's just fine: that module only exists for integration with lazify. But if a module has a reason to exist on its own, then it should also be tested on its own, as much as reasonable.

[Technologicat]

Now at 92%.

Basically unpythonic.syntax.letdoutil and unpythonic.syntax.simplelet are the last ones that are easy-ish (but somewhat tedious) to get fully covered. Then it's just the REPL/networking and test framework code, and meta-utilities such as setup.py and countlines.py.

[Technologicat]

Now at 92.67%.

That's it for the tedious but low-hanging fruit.

Now just to layer the tests (issue #69) so that we can test unpythonic.syntax.testingtools, and then leave adding tests for the REPL/networking code to a future release (since this is already much better, and this release is already getting big).

[Technologicat]

Ah, but before that:

• Go through all tests one more time, and:
  • Use the[] as appropriate to auto-print something useful for debugging if the tests ever fail
  • Measure per-unit coverage, improve where necessary
  • Done for regular code as of 66fd6dd.
  • Done for macro code as of c2577d0.
• Figure out and add a test for the corner case in continuations.

In the future:

• Improve naming in all test examples (no foo, bar).

[Technologicat]

As of 0.15.0-pre, we're at 94% coverage.