Continuous Integration

[certik]

What I have found is that the notebooks break in a year or two, since I might have different versions of Python packages installed, the upstream data becomes unavailable, or some other reason. Also, you want to make sure that if people send PRs, that you can safely merge them.

Part of the solution is to actually test the notebooks themselves on Travis, as long as they run in reasonable time (say 30 min or less). That way it tests a particular version of packages, so if somebody sends a PR and it breaks in an unrelated notebook, it's easier to debug. At the very least you can run the notebook using the nbviewer command line tools. But it'd be nice to also test that they actually work --- I don't have a good solution for this.

The other problem is that the website where I got the data from changes and I can no longer download it, and then my whole pipeline that depended on the exact format and so on is useless. I don't have a good solution for this, since typically you don't own the copyright to the data, so you can't just upload it yourself somewhere. Also it can be big.