YangJie

I wonder if we have enough UTs to check the change of `kinesis` related code

https://github.com/apache/spark/pull/37605#issuecomment-1221814039 cc @HyukjinKwon I suspect the flaky test is caused by machine load. After the timeout is increased, there is no failure yet, let's test and check more times

I'm not sure if my GA can't reproduce the issue

> Hm, I doubt if the timeout is the actual problem. This test doesn't take too much time if you run locally actually. OK, let me see how to get...

cc @mridulm @tgravescs and @squito for further discussion, the previous thread is here https://github.com/apache/spark/pull/37648#discussion_r963271711

GA failed not related to current pr, need wait https://github.com/apache/spark/pull/37815

friendly ping @sunchao, do you know `MiniYARNCluster` can start with `YarnConfiguration.NM_RECOVERY_ENABLED = true`? I try to set this to YarnConfig, but `MiniYARNCluster` start failed: ``` 2022-09-10T11:44:42.1710230Z Cause: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.org.iq80.leveldb.DBException 2022-09-10T11:44:42.1715234Z...

https://github.com/apache/spark/blob/bf5103ae5bb938102850160048e2a1656e648244/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnShuffleIntegrationSuite.scala#L46-L54 add `yarnConfig.setBoolean(YarnConfiguration.NM_RECOVERY_ENABLED, true)` to above `newYarnConfig` method, and test with hadoop-2.7 ``` build/mvn clean install -pl resource-managers/yarn -Pyarn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnShuffleIntegrationWithLevelDBBackendSuite -Phadoop-2 ``` then failed as follows: ``` YarnShuffleIntegrationWithLevelDBBackendSuite: org.apache.spark.deploy.yarn.YarnShuffleIntegrationWithLevelDBBackendSuite...

So I think `assert(!ShuffleTestAccessor.reloadRegisteredExecutors(dbBackend, execStateCopy).isEmpty)` and the relevant code is an unreachable code currently... @tgravescs @mridulm

thanks @sunchao