Investigate ElasticSearch EBS Volume Usage

[timckt]

Background

Live ElasticSearch seems to have some problem on the storage usage.

We bumped the elasticsearch ebs volume from 3TB to 4TB on 2024-06-17 which is 2 weeks ago as of ticket creation.

We got the alert again for FreeStorageSpaceTooLow since 2024-06-26 22:53.

The time before last was 2023-08-11 which is 10 months ago, bumping the EBS size from 1.5TB to 3TB.

Find the reason why is the storage consumed so fast.

Proposed user journey

Approach

Which part of the user docs does this impact

Communicate changes

• [ ] post for #cloud-platform-update
• [ ] Weeknotes item
• [ ] Show the Thing/P&A All Hands/User CoP
• [ ] Announcements channel

Questions / Assumptions

Definition of done

• [ ] readme has been updated
• [ ] user docs have been updated
• [ ] another team member has reviewed
• [ ] smoke tests are green
• [ ] prepare demo for the team

Reference

How to write good user stories