[BUG] DecommissionControllerTests.testTimesOut flaky

[dblock]

Describe the bug

https://github.com/opensearch-project/OpenSearch/pull/5191 https://build.ci.opensearch.org/job/gradle-check/6747/

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.cluster.decommission.DecommissionControllerTests.testTimesOut" -Dtests.seed=AF2F88C09A211CBA -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-ZA -Dtests.timezone=Asia/Ashgabat -Druntime.java=17

org.opensearch.cluster.decommission.DecommissionControllerTests > testTimesOut FAILED
    java.lang.AssertionError
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertTrue(Assert.java:53)
        at org.opensearch.cluster.decommission.DecommissionControllerTests.testTimesOut(DecommissionControllerTests.java:234)

[andrross]

This was fixed but not backported: #4683

I'll do the backport now.

[peternied]

• Reproduced in https://github.com/opensearch-project/OpenSearch/pull/11957, see logs

java.lang.AssertionError: Expected onFailure to be called
Expected: not null
     but: was null
	at __randomizedtesting.SeedInfo.seed([95FCFD6AF5DCEFB:6FAB526F364153D7]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.opensearch.cluster.decommission.DecommissionControllerTests.testTimesOut(DecommissionControllerTests.java:196)
...

Repro instruction

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.cluster.decommission.DecommissionControllerTests.testTimesOut" -Dtests.seed=95FCFD6AF5DCEFB -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-SA -Dtests.timezone=America/Puerto_Rico -Druntime.java=21

[Pranshu-S]

Taking a look.