timeout on travis ci with --cov --nbval-lax --cov-append

[MarvinT]

trafficstars

I have been having inconsistent errors recently, however, this most recent one seems to have stuck. It might be related to the number of collected pytest items because otherwise, nothing has impacted the code that it seems to be getting stuck on (253 items on this example failing build). It also seems to be dependent on python version, but that could be for quite a few reasons. Below are py3.6 examples.

Here's an example of failing build: https://travis-ci.com/MarvinT/morphs/jobs/201281071 which also includes the commands to recreate the env and run the tests.

It crashes (10 minute timeout on travis CI) on a cell that takes seconds to run normally.

I've had these inconsistent issues before and I "solved" it by running the notebooks separately from the main tests.

This is the command that is getting run that eventually crashes:

pytest notebooks/ --cov=morphs --nbval-lax --current-env --cov-append

I'm going to try to split it up into smaller groups of notebooks here so if this passes that might be a temporary solution: https://travis-ci.com/MarvinT/morphs/jobs/201285641

[takluyver]

Is the cell that it crashes on doing anything specific? Some operations can be much slower on Travis than on a normal system, though a few seconds to 10 minutes would be an extreme slowdown.

[MarvinT]

The cell that is crashing is

all_dims = np.sort([l + g for l in spects for g in spects[l]])

where l and g are characters a-h and there are a total of 24 combinations. It takes 4 ms on my machine.

I originally thought it was the cells after where I plot (imshow) a couple of matrices which take 3 seconds each. Does nbval have the same issue as jupyter where it doesn't necessarily report cells complete when they are actually completed?

[takluyver]

It might be dependent on the kernel reporting when the output from a cell is finished, but the IPython kernel should do that correctly.

I don't know why that code should cause issues for nbval, because it shouldn't produce any output, and basically all the special behaviour in nbval is around checking output.

Have you run the same pytest command with nbval locally?

[MarvinT]

It runs fine locally and it ran fine on travis in the past. None of the changes I made affected that notebook. Even after splitting up the tests into each notebook individually it was crashing. I ended up adding #Nbval_skip to the two cells after it that take 3 seconds each and it ended up passing. Sorry, I'm not able to debug this more atm, but I did want to let you guys know that there were some strange performance issues with some combination of these flags, and also that I was able to get all these flags to work together.

currently my .travis.yml contains:

script:
  - pytest -v testing/ --cov=morphs 
  - pytest -v notebooks/1.* --cov=morphs --nbval-lax --current-env --cov-append
  - pytest -v notebooks/2.* --cov=morphs --nbval-lax --current-env --cov-append
  # - pytest -v notebooks/3.* --cov=morphs --nbval-lax --current-env --cov-append
  - pytest -v notebooks/4.* --cov=morphs --nbval-lax --current-env --cov-append
  - pytest -v notebooks/5.* --cov=morphs --nbval-lax --current-env --cov-append
  - pytest -v notebooks/6.* --cov=morphs --nbval-lax --current-env --cov-append
  - pytest -v notebooks/7.* --cov=morphs --nbval-lax --current-env --cov-append
  - pytest -v notebooks/8.* --cov=morphs --nbval-lax --current-env --cov-append
  - pytest -v notebooks/supplemental --cov=morphs --nbval-lax --current-env --cov-append