K8SPXC-1411: allow to enable/disable TLS in a running cluster

[pooknull]

https://perconadev.atlassian.net/browse/K8SPXC-1411

DESCRIPTION

This PR allows operator to enable/disable TLS in a running cluster by automating the following tasks:

when .spec.tls.enabled is switched to false:

1. patch .spec.pause to true
2. wait until all pods are deleted
3. patch spec.unsafeFlags.tls to true
4. delete TLS secrets
5. patch .spec.pause to false

when .spec.tls.enabled is switched to true:

1. patch .spec.pause to true
2. wait until all pods are deleted
3. patch spec.unsafeFlags.tls to false
4. patch .spec.pause to false

A percona.com/tls annotation has also been added to a cluster. It will show the state of .spec.tls.enabled field before it was switched. The values of these annotations are enabled and disabled values. After all automated tasks have been completed, it will be updated with the actual state of .spec.tls.enabled.

Note to developers: the deploy method contained a lot of duplicated code from the updatePod method. In this PR I decided to minimize it by using the updatePod inside.

CHECKLIST

Jira

• [x] Is the Jira ticket created and referenced properly?
• [x] Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• [x] Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• [x] Is an E2E test/test case added for the new feature/change?
• [ ] Are unit tests added where appropriate?
• [ ] Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

• [x] Are all needed new/changed options added to default YAML files?
• [x] Are all needed new/changed options added to the Helm Chart?
• [x] Did we add proper logging messages for operator actions?
• [x] Did we ensure compatibility with the previous version or cluster upgrade process?
• [x] Does the change support oldest and newest supported PXC version?
• [x] Does the change support oldest and newest supported Kubernetes version?

[JNKPercona]

Test name	Status
affinity-8-0	passed
auto-tuning-8-0	passed
cross-site-8-0	passed
custom-users-8-0	passed
demand-backup-cloud-8-0	passed
demand-backup-encrypted-with-tls-8-0	passed
demand-backup-8-0	passed
haproxy-5-7	passed
haproxy-8-0	passed
init-deploy-5-7	passed
init-deploy-8-0	passed
limits-8-0	passed
monitoring-2-0-8-0	passed
one-pod-5-7	passed
one-pod-8-0	passed
pitr-8-0	passed
pitr-gap-errors-8-0	passed
proxy-protocol-8-0	passed
proxysql-sidecar-res-limits-8-0	passed
pvc-resize-5-7	passed
pvc-resize-8-0	passed
recreate-8-0	passed
restore-to-encrypted-cluster-8-0	passed
scaling-proxysql-8-0	passed
scaling-8-0	passed
scheduled-backup-5-7	passed
scheduled-backup-8-0	passed
security-context-8-0	passed
smart-update1-8-0	passed
smart-update2-8-0	passed
storage-8-0	passed
tls-issue-cert-manager-ref-8-0	passed
tls-issue-cert-manager-8-0	passed
tls-issue-self-8-0	passed
upgrade-consistency-8-0	passed
upgrade-haproxy-5-7	passed
upgrade-haproxy-8-0	passed
upgrade-proxysql-5-7	passed
upgrade-proxysql-8-0	passed
users-5-7	passed
users-8-0	passed
validation-hook-8-0	passed
We run 42 out of 42

commit: https://github.com/percona/percona-xtradb-cluster-operator/pull/1844/commits/eeb6ea9d34b8e934b88f6246148d9988280352fc image: perconalab/percona-xtradb-cluster-operator:PR-1844-eeb6ea9d