Jenkinsfiles should assert that the rollout to Staging/Production succeeds

[jstrachan]

Right now a Pipeline could fail to get a new version of a pod running in an enviroment (e.g. the pod never becomes ready - maybe due to quota issues or a missing environment specific Service, Secret or ConfigMap or something.

Currently once the apply is done, the kubernetesApply() just assumes everything's great and carries on.

It would be nice to have a better flavour of this which does the Arquillian equivalent of this line:

             assertThat(kubernetesClient).deployments().pods().isPodReadyForPeriod();

Then the pipeline would wait for the pods to go green & be ready (readiness checks + liveness checks kick in) - if things don't work it'd barf the build.

Maybe extra bonus points would be to automatically rollback the Deployment change if the new version doesn't startup correctly?

[rawlingsj]

Maybe extra bonus points would be to automatically rollback the Deployment change if the new version doesn't startup correctly?

Would it be easy to leave a failed pod hanging around so folks can go and look at the failed deployment?

[iocanel]

Can't we configure that as part of the environment?

[jstrachan]

maybe there should be a standard timeout to kubernetesApply() where it waits for any Deployments / RC / RS resources to go ready (have a deployment.status.available > 0) within a time period?

e.g. for each resource being applied, if its a Deployment/RC/RS and the replicas > 0 then wait up to a timeout period for the status.available to be > 0 and status.unavailable == 0 or fail the build.

I started doing something similar in gofabric8 wait-for recently: https://github.com/fabric8io/gofabric8/blob/master/cmds/wait_for.go#L245

[jstrachan]

kubernetesApply() now takes an optional readinessTimeout - we just need to find that value (maybe having a default OOTB value we set on the fabric8-pipeline ConfigMap or on a per environment basis in fabric8-environments? that we then pass into the kubernetesApply() call.