[CI] org.elasticsearch.cluster.service.MasterService.BatchingTaskQueue#submitTask throws unexpected EsRejectedExecutionException

[DaveCTurner]

Probably doesn't reproduce directly but the problem is clear from the stack trace.

Build scan: https://gradle-enterprise.elastic.co/s/uqld27x6h52bo/tests/:x-pack:plugin:downsample:internalClusterTest/org.elasticsearch.xpack.downsample.DataStreamLifecycleDownsampleDisruptionIT/testDataStreamLifecycleDownsampleRollingRestart

Reproduction line:

./gradlew ':x-pack:plugin:downsample:internalClusterTest' --tests "org.elasticsearch.xpack.downsample.DataStreamLifecycleDownsampleDisruptionIT.testDataStreamLifecycleDownsampleRollingRestart" -Dtests.seed=4FFA60CD48D62773 -Dtests.locale=ar-KW -Dtests.timezone=America/Bahia -Druntime.java=21

Applicable branches: main

Reproduces locally?: Yes

Failure history: Failure dashboard for org.elasticsearch.xpack.downsample.DataStreamLifecycleDownsampleDisruptionIT#testDataStreamLifecycleDownsampleRollingRestart

Failure excerpt:

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4d620954[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@1278403[Wrapped task = [threaded] master service timeout handler for [cluster_health (wait_for_events [LANGUID])][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction$2@43ab3e6d] after [30s]]] on org.elasticsearch.threadpool.Scheduler$SafeScheduledThreadPoolExecutor@78f3f09c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 156] (shutdown)

  at __randomizedtesting.SeedInfo.seed([4FFA60CD48D62773:9863F77D1EDBC23D]:0)
  at org.elasticsearch.common.util.concurrent.EsRejectedExecutionHandler.newRejectedException(EsRejectedExecutionHandler.java:51)
  at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:35)
  at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
  at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
  at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
  at org.elasticsearch.threadpool.ThreadPool.schedule(ThreadPool.java:576)
  at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue.submitTask(MasterService.java:1525)
  at org.elasticsearch.cluster.service.MasterService.submitUnbatchedStateUpdateTask(MasterService.java:537)
  at org.elasticsearch.cluster.service.ClusterService.submitUnbatchedStateUpdateTask(ClusterService.java:229)
  at org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction.submitUnbatchedTask(TransportClusterHealthAction.java:221)
  at org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction.waitForEventsAndExecuteHealth(TransportClusterHealthAction.java:162)
  at org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction.masterOperation(TransportClusterHealthAction.java:101)
  at org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction.masterOperation(TransportClusterHealthAction.java:49)
  at org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:125)
  at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:236)
  at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)
  at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  at java.lang.Thread.run(Thread.java:1583)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)