monasca
Fix Travis build issues
Our builds are logging way too much and taking too long. https://travis-ci.org/monasca/monasca-docker/builds/273282923 ran for 7 hours (!!!) before being terminated because the log output exceeded 4 MiB. I don't think it's acceptable to abuse Travis's free resources like this, even if accidental.
A few short-term fixes are needed:
- [x] Fix timothyb89/dbuild#3
- [x] Put a hard limit on the max number of modules to build, probably ~5 is reasonable
- [x] Put log output somewhere else. Saving to a GCP bucket is probably a good choice, plus we can split each container's logs for easier reading.
Long term we should also rethink a few things:
- Consider running Tempest tests using a Travis cron job instead of with every build: https://docs.travis-ci.com/user/cron-jobs/
- we can have the script file an issue when the build fails
- Add a max build time restriction for each container build
- we want to keep this fairly low, say 30 minutes each and 60 minutes total
- ~~Consider alternative testing strategies so we don't end up in situations with 19 modules being built at once for minor fixes. Not sure what solutions here would be, if any.~~ Fixed in #209
I just pushed https://github.com/timothyb89/dbuild/commit/4a9b6106963e803d5b483749bfb8779bb5108f81 as a workaround for timothyb89/dbuild#3. This has the immediate effect of removing container build logs from Travis, so we'll want to additionally turn on --build-log-dir=... when running dbuild and then upload logs somewhere at the end of CI jobs.
Module limit was added in https://github.com/monasca/monasca-docker/pull/209, and https://github.com/monasca/monasca-docker/pull/211 stores logs in a GCS bucket.
