Fix flaky test suite

[daneah]

Expected behavior I expect the test suite to run reliably and give strong signal about whether something is functionally broken or not.

Actual behavior The tests regularly fail, maybe 75% of the time, for something that does not fail upon rerun of the job. This ends up causing lost time pretty often.