alerting icon indicating copy to clipboard operation
alerting copied to clipboard

[BUG] Alerts break and alerting lock index becomes unrecoverable

Open rkbennett opened this issue 1 year ago • 1 comments

Describe the bug

When using alerting the .opensearch-alerting-config-lock shard isn't rotated and can grow past a recoverable shard size which then breaks all alerting. Because this is a shard that isn't managed by the user, a rotation strategy should be default

Related component

Storage

To Reproduce

  1. Create an alert
  2. Restart the cluster once the shard size has grown past the limit_size_in_bytes setting
  3. .opensearch-alerting-config-lock shard will never recover and alerts will no longer work

Expected behavior

System shards shouldn't get into permanently broken states

Additional Details

Plugins Security

Host/Environment (please complete the following information):

  • OS: Opensearch 2.18 container image
  • Version 2.18

rkbennett avatar Nov 28 '24 13:11 rkbennett

[Catch All Triage - 1, 2, 3]

dblock avatar Dec 16 '24 17:12 dblock