Add support for automatically collecting and uploading core dumps

[jlebon]

E.g. a collect-cores key? This would require different approaches depending on the environment:

• Virtualized: enable core dumping to e.g. /var/tmp/coredumps during provisioning, and just upload all the cores found at the end of the test run.

• Containerized: in this scenario, rhci itself is running either in a privileged container or directly on the host. Either way, it should be able to write to core_pattern. One issue is linking back core dumps with the right container/test run for upload. Maybe we should just make our own |/usr/bin/rhci-coredump %p handler and teach it to cross link PIDs with containers? Though not sure how that would work if rhci itself is containerized. Relying on just the PID in the final core dump name would have races.

We would also need to upload the binaries, of course. Maybe we can start with a limit of 10 MB total (for core + binaries) and see how useful it is.

[cgwalters]

See also https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2016-September/msg00044.html

This type of thing is exactly why I think we should be trying to use Kube/OpenShift as our backend, since any improvements we make for core collection in CI testing will also apply to production workloads.

[jlebon]

See also https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2017-March/msg00053.html.