cloud-platform icon indicating copy to clipboard operation
cloud-platform copied to clipboard

Implement longer term storage for our logs

Open razvan-moj opened this issue 2 years ago • 0 comments

Background

We have 30 days of logs, and want 13 months. AWS meeting recommended implementing Warm and Cold storage for the ES cluster.

Approach

  • instances we use are quite old, change to newer. Might be useful to upgrade the instance type, as storage not CPU is our main cost. (done)
  • upgrade release to at least ES 7.9 (done)
  • add the options of UltraWarm and Cold

Which part of the user docs does this impact

The process should be completely transparent for our users

Definition of done

  • [ ] cluster upgraded
  • [ ] newer instance types
  • [ ] another team member has reviewed
  • [ ] logs still come in
  • [ ] curator job edited to cleanup after 1yr
  • [ ] infra terraform code edited for the new config

Note: scope of this is not to fix the json parser, that is the topic of a separate story

razvan-moj avatar Jul 19 '22 14:07 razvan-moj