Mary Gouseti

From a quick look it looks like the test is failing because the shrink index name is `null`. What the test does is: - Set-up a policy with the wrong...

Based on the logging ILM explained that is being logged: ``` [2022-07-28T15:45:56,690][INFO ][o.e.x.TimeSeriesRestDriver] [testAutomaticRetryFailedShrinkAction] --> index index-xsvudpmsaf, explain {index-xsvudpmsaf={index=index-xsvudpmsaf, managed=true, policy=policy-omzmg, index_creation_date_millis=1659019511781, time_since_index_creation=44.9s, lifecycle_date_millis=1659019511781, age=44.9s, phase=warm, phase_time_millis=1659019517402, action=shrink, action_time_millis=1659019519622, step=check-target-shards-count,...

- First attempt with invalid target shard count: ``` [2022-07-29T22:57:19,024][INFO ][o.e.x.i.IndexLifecycleRunner] [javaRestTest-0] [index-oewyaxaocl] State changed for index [2022-07-29T22:57:19,024][TRACE][o.e.x.i.IndexLifecycleRunner] [javaRestTest-0] [index-oewyaxaocl] retrieved current step key: {"phase":"warm","action":"shrink","name":"check-target-shards-count"} [2022-07-29T22:57:19,042][TRACE][o.e.x.i.PolicyStepsRegistry] [javaRestTest-0] parsed steps for...

So the culprit is here: ``` if (busyIndices.contains(Tuple.tuple(indexMetadata.getIndex(), currentStepKey))) { // try later again, already doing work for this index at this step, no need to check for more work...

There is a hint for this one :). In `org.elasticsearch.xpack.ilm.IndexLifecycleRunner`: ``` /** * Tracks already executing {@link IndexLifecycleClusterStateUpdateTask} tasks in {@link #executingTasks} to prevent queueing up * duplicate cluster state...

Relates to https://github.com/elastic/elasticsearch/pull/78390

See https://github.com/elastic/elasticsearch/pull/89176 for the latest update.

I am afraid this happened again, see https://gradle-enterprise.elastic.co/s/cwlcjoocvaehg/tests/:server:test/org.elasticsearch.cluster.coordination.CoordinatorTests/testSingleNodeDiscoveryStabilisesEvenWhenDisrupted?top-execution=1. I am reopening this issue, if this is not the right thing to do, let me know and I will create a...

@DaveCTurner I do agree that we need something a bit more complex than simply alerting on the watermarks. I would say that the golden rule of alerting is that it...

**Requirement update** We took into consideration all the concerns above and we came up with the following requirements which we believe is a good balance of reduced complexity and usefulness....