PWX-37204 , PWX- 37178 : Resolving gaps in KVDB fix

[svijaykumar-px]

What this PR does / why we need it: This PR contains the fixes for 2 tickets :

1. PWX-37178 - Ensuring that atleast 3 kvdb nodes are present to upgrade those portworx nodes which have internal kvdb running in it. This is necessary as if a kvdb node is down for more than 2 minutes it gets removed from the list of kvdb nodes thus leading to confusion that kvdb cluster is working correctly. Adding extra check to ensure 3 nodes are present along with all 3 being healthy to upgrade kvdb nodes.
2. PWX-37204 - When portworx is installed freshly, the portworx sdk server is not up for a while. This causes the getKVDBNodeAvailability function to return error and pause the rolling update and make cluster to enter degraded state till portworx sdk starts. The fix ensures that this error is ignored in the case of fresh install of portworx .

Which issue(s) this PR fixes (optional) Closes # PWX-37204, PWX-37178

Testing notes: Tested fresh install, cluster no longer goes to degraded state Tested with 5 node cluster, kvdb down on one node and wait for 3 minutes, then start the upgrade. The other 2 kvdb nodes do not get upgraded

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 75.99%. Comparing base (9854899) to head (18e4583).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1542   +/-   ##
=======================================
  Coverage   75.99%   75.99%           
=======================================
  Files          76       76           
  Lines       20205    20208    +3     
=======================================
+ Hits        15354    15357    +3     
  Misses       3792     3792           
  Partials     1059     1059           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.