alerting
alerting copied to clipboard
[BUG] Alerts break and alerting lock index becomes unrecoverable
Describe the bug
When using alerting the .opensearch-alerting-config-lock shard isn't rotated and can grow past a recoverable shard size which then breaks all alerting. Because this is a shard that isn't managed by the user, a rotation strategy should be default
Related component
Storage
To Reproduce
- Create an alert
- Restart the cluster once the shard size has grown past the limit_size_in_bytes setting
- .opensearch-alerting-config-lock shard will never recover and alerts will no longer work
Expected behavior
System shards shouldn't get into permanently broken states
Additional Details
Plugins Security
Host/Environment (please complete the following information):
- OS: Opensearch 2.18 container image
- Version 2.18
[Catch All Triage - 1, 2, 3]