multi-node integration failing for Windows CI for 1.3 branch

[amitgalitz]

What is the bug? Multi node integration tests are failing for 1.3 branch on windows. This is due to an issue deleting the information for the previous cluster that was spun up. Running multi node integration works fine manually on windows however there is probably some issue with build.gradle or core setup for deleting old cluster info.

* What went wrong:
Execution failed for task ':integTest'.
> java.io.IOException: Unable to delete directory 'D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0'
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
    Failed to delete some children. This might happen because a process has files open or has its working directory set in the target directory.
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\HdrHistogram-2.1.9.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\hppc-0.8.1.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\jackson-core-2.14.1.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\jackson-dataformat-cbor-2.14.1.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\jackson-dataformat-smile-2.14.1.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\jackson-dataformat-yaml-2.14.1.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\java-version-checker-1.3.7-SNAPSHOT.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\jna-5.5.0.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\joda-time-2.10.12.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\jopt-simple-5.0.4.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\jts-core-1.15.0.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\log4j-api-2.17.1.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\log4j-core-2.17.1.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\lucene-analyzers-common-8.10.1.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\lucene-backward-codecs-8.10.1.jar
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\lib\lucene-core-8.10.1.jar
    - and more ...
    New files were found. This might happen because a process is still writing to the target directory.
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE
    - D:\a\anomaly-detection\anomaly-detection\build\testclusters\integTest-0\distro\1.3.7-ARCHIVE\logs\gc.log
   

How can one reproduce the bug? Steps to reproduce the behavior: on windows machine, 1.3 branch: ./gradlew integTest -PnumNodes=3

https://github.com/opensearch-project/anomaly-detection/actions/runs/3623131490/jobs/6109210683

[kaituo]

found it is due to gradle test failed to kill the java process while existing. The fix is to kill the process manually first (look for Java process) before issuing another build or test command.