speed up build by skipping tests unaffected by changed code

[notestaff]

Could try one of https://pypi.python.org/pypi/pytest-testmon https://pypi.python.org/pypi/pytest-incremental https://pypi.python.org/pypi/pytest-cov-exclude https://pypi.python.org/pypi/pytest-cagoule https://pypi.python.org/pypi/pytest-knows https://pypi.python.org/pypi/pytest-skipper https://github.com/ChrisBeaumont/smother together with https://docs.travis-ci.com/user/caching/

Would need to pass cached coverage info for skipped tests to coveralls, so that skipped tests don't cause a drop in coverage.

(also, could turn on https://docs.travis-ci.com/user/caching/#pip-cache -- or is that already done by caching $HOME/miniconda ?)

[notestaff]

@tomkinsc @dpark01

from @yesimon "The problem with drops in coveralls coverage after updating a branch is because I made a change in the tests to only run unit versus integration tests for travis branch versus pull request tests.

This is precisely to deduplicate testing because each pull request generates both a branch test run and pull request merge test run

Right now I believe coverage is based off of whichever report is submitted last, which means that the slower test is usually the final result which is a problem with certain updates IIRC if a test doesn't get run.

[yesimon]

The problem is that these all require some mechanism of storing state, either in json or sqlite db or other format. We don't have any mechanism of storing state except in travis cache. Unfortunately travis cache is not split for each directory, but rather it is one large tarball that contains everything. This means that using one of these plugins will basically cause cache invalidation every time and is costly now that we're actually using the cache properly.

It is still possible if you store the state elsewhere, like a heroku server, but it will take a decent amount of work to build the microservice.

[notestaff]

@yesimon We needn't update the cache every time. If no state is stored for any tests, we store it for all the tests. We use stored state to skip tests, but we don't add to the stored state. So new tests added since we've stored the state will never be skipped, but most tests will be. Periodically, we can rebuild the cache to store the state for the new tests, but that can be done infrequently.

[notestaff]

Btw, one can check whether the list of files relevant to a test was correctly determined: just erase all others, and re-run the test. (Or, more reasonably, link just the relevant files into a separate dir, and run the test there.)

Also, can get the relevant-files list without trusting python's trace/coverage machinery (which only tracks python files anyway): find files whose atime is later than the test start time.