HDDS-3419. Throw exception when healthy nodes are not sufficient

[pratapchandu]

What changes were proposed in this pull request?

Fix and enable test cases in TestRatisPipelineCreateAndDestroy. Throw exception when available healthy nodes are not sufficient

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3419

How was this patch tested?

All tests in TestRatisPipelineCreateAndDestroy were running successfully with the fix on developers machine. build-branch was successful with the change https://github.com/pratapchandu/ozone/actions/runs/3264534699

[kerneltime]

cc @devmadhuu @neils-dev @aswinshakil

[kerneltime]

Can you add some more explanation as to why adding the exception helped the test cases pass reliably?

[pratapchandu]

@neils-dev , updated code it throws Exception with result code FAILED_TO_FIND_HEALTHY_NODES when no healthy node is available. changed test to accept FAILED_TO_FIND_HEALTHY_NODES.

[pratapchandu]

Thanks @neils-dev, @kerneltime, @devmadhuu, @aswinshakil for reviews. It passed all checks after re-run.

[adoroszlai]

Unfortunately the PR only fixed one test case but enabled the whole test class. Now there are intermittent failures in the test cases that were not updated:

• https://github.com/adoroszlai/ozone-build-results/blob/master/2022/11/08/18312/it-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestroy.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2022/11/08/18321/it-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestroy.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2022/11/08/18331/it-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestroy.txt