feat(esClientDrain): enhance Drain ES Client function

[s-vkropotko]

One-line summary

Update Drain functionality

Description

Add an ability to set parameters for retry logic with Resty lib. Add handling for not finished shards migration.

About shards migration and Drain behavior

We have a kinda bug in waitForEmptyEsNode function. In this function, after we finish a retry logic we don't check the response and proceed with the deletion of the pod but it can be wrong (shards can be still on the pod). So validation for this case is added.

Also, an additional logic added withremoveFromExcludeIPList. So if checking of remaining shards finishes without success now we remove the pod from the exclude list in ES cluster settings (before, it left in the excluded list).

There is the next idea behind it - ES-Op can change scaling direction after waiting for condition(migration of remaining shards from Pod) so it looks better to start with the state before Drain was started.

Types of Changes

What types of changes does your code introduce? Keep the ones that apply:

• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)

Review

List of tasks the reviewer must do to review the PR

• [x] Tests

Deployment Notes

To test this behavior ES-Op should be deployed with small values of retry logic options:

        - --esclient-retry-count=2
        - --esclient-retry-waittime=1s
        - --esclient-retry-max-waittime=10s

[otrosien]

If I understand the PR correctly, it should fix this bug: https://github.com/zalando-incubator/es-operator/issues/128

[s-vkropotko]

#128 Yes, I think so. Sorry, I didn't check the list of existed issues 😃

[otrosien]

👍

[otrosien]

@mikkeloscar mind to also take a look?

[otrosien]

I would like to take another look at this.