[CI] Parallel Integration Test Desgin

[simon-mo]

I’m starting to work on overhauling our integration test. Our currrent integration test is slow and flaky.

We have the following limitation and requirement:

• We can run only one Jenkins Job because amplab Jenkins is quite outdated. And the lab won’t update Jenkins until the end of year.
• We need to run test on kubernetes. We used to run tests on EKS, this is costly. There are minikube installed on Jenkins workers so we can utilize that.
• We need to run GPU test. There is one GPU worker. This has to be a separate Jenkins Job that is going to run after the main Job.
• We need to parallellize and stabilize tests. This can be done by splitting tests further into smaller parts and run each of them in docker container. We can parallelize docker container by using a Makefile to run them (make -j)

Here’s things I’ll do to improve the experience:

• Enable logging driver for docker container. Running dockers in parallel will clobber the output log on Jenkins. We will ask docker to send the logs to a fluentd container. Fluentd will send logs to a remote server.
  • [ ] Setup Kafka
  • [ ] Setup fluentd
• I’ll maintain this server to show disaggregated logs per container in real time.
  • [x] Log display server backbone
  • [ ] Log UI refinement
• Writing Makefile for fine-grained dependency will be error-prone. We will use a python script to generate Makefile.
  • [x] Generate build make file
  • [ ] Generate test make file
• Use travis for minikube.
  • [x] Install travis
  • [ ] Porting tests to use minikube

Lastly:

• Should we deprecate JVM support? Yes