FIREBREAK: Monitor EKS using GCP Stackdriver

[davidread]

Trying out using GCP Stackdriver to monitor metrics and logs of our EKS cluster.

Background

We currently use Promtheus for monitoring and Amazon ES for logging, and are looking at other options.

Why a non-AWS tool? If AWS is having service issues, then there's an advantage in having the logging & metrics done from outside AWS. Service issues tend not to be black and white - there is often a partial failure, so having solid observability. During the issues, even if collection of these things cannot continue, the tool would still provide access to the lead-up period and provide querying functions.

Why GCP Stackdriver? We've tried some other Saas observability tools, and there is some hesitancy over the enterprise-style pricing. We're hopeful that in common with other cloud services, Stackdriver pricing is transparent and PAYG, to include in the comparison.

At our workshop on Resiliency against AWS service issues we agreed that this is an avenue we should explore

Questions / Assumptions

Definition of done

• [ ] Firebreak finding documented appropriately
• [ ] Demo completed
• [ ] Decision made on whether to progress Firebreak work
• [ ] Firebreak next step Issues created
• [ ] New Issues referenced in this story before closure

[pwyborn]

https://bluemedora.com/monitoring-amazon-eks-with-stackdriver-blue-medora/

[pwyborn]

As per standup conversation today 19/04/2022 - Do we have GCP account yet?