K8SPS-505 fix demand-backup-retry test robustness

[valmiranogueira]

CHANGE DESCRIPTION

Problem:

• Ignore kubectl stderr with stream failures, but keep the mysql stderr for debugging when running a query
• Increase cluster data to avoid race condition while killing the pod and waiting expected backup failure
• Increase backup failure timeout to 2 minutes
• With the loaded data it's necessary to increase default max_allowed_packet to perform data validation
• Removed backoff configuration for backups, since the pod that is killed may take time to start back and 3 retries could be not enough

Cause: Short explanation of the root cause of the issue if applicable.

Solution: Short explanation of the solution we are providing with this PR.

CHECKLIST

Jira

• [ ] Is the Jira ticket created and referenced properly?
• [ ] Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• [ ] Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• [ ] Is an E2E test/test case added for the new feature/change?
• [ ] Are unit tests added where appropriate?

Config/Logging/Testability

• [ ] Are all needed new/changed options added to default YAML files?
• [ ] Are all needed new/changed options added to the Helm Chart?
• [ ] Did we add proper logging messages for operator actions?
• [ ] Did we ensure compatibility with the previous version or cluster upgrade process?
• [ ] Does the change support oldest and newest supported PS version?
• [ ] Does the change support oldest and newest supported Kubernetes version?

[JNKPercona]

Test Name	Result	Time
async-ignore-annotations-8-4	passed	00:06:32
async-global-metadata-8-4	passed	00:15:24
async-upgrade-8-0	passed	00:12:20
async-upgrade-8-4	passed	00:12:37
auto-config-8-4	passed	00:25:22
config-8-4	passed	00:16:53
config-router-8-0	passed	00:07:11
config-router-8-4	passed	00:07:36
demand-backup-minio-8-0	passed	00:19:26
demand-backup-minio-8-4	passed	00:19:54
demand-backup-cloud-8-4	passed	00:21:17
demand-backup-retry-8-4	passed	00:15:55
async-data-at-rest-encryption-8-0	passed	00:14:37
async-data-at-rest-encryption-8-4	passed	00:14:08
gr-global-metadata-8-4	passed	00:14:31
gr-data-at-rest-encryption-8-0	passed	00:13:24
gr-data-at-rest-encryption-8-4	passed	00:15:12
gr-demand-backup-minio-8-4	passed	00:12:20
gr-demand-backup-cloud-8-4	passed	00:22:29
gr-demand-backup-haproxy-8-4	passed	00:10:12
gr-finalizer-8-4	passed	00:05:42
gr-haproxy-8-0	passed	00:04:24
gr-haproxy-8-4	passed	00:04:09
gr-ignore-annotations-8-4	passed	00:04:51
gr-init-deploy-8-0	passed	00:09:05
gr-init-deploy-8-4	passed	00:09:33
gr-one-pod-8-4	passed	00:05:55
gr-recreate-8-4	passed	00:17:26
gr-scaling-8-4	passed	00:07:45
gr-scheduled-backup-8-4	passed	00:19:29
gr-security-context-8-4	passed	00:10:03
gr-self-healing-8-4	passed	00:20:57
gr-tls-cert-manager-8-4	passed	00:09:10
gr-users-8-4	passed	00:05:23
gr-upgrade-8-0	passed	00:09:31
gr-upgrade-8-4	passed	00:09:31
haproxy-8-0	passed	00:09:01
haproxy-8-4	passed	00:08:20
init-deploy-8-0	passed	00:06:37
init-deploy-8-4	passed	00:05:40
limits-8-4	passed	00:07:05
monitoring-8-4	passed	00:14:03
one-pod-8-0	passed	00:05:54
one-pod-8-4	passed	00:05:32
operator-self-healing-8-4	passed	00:12:03
pvc-resize-8-4	passed	00:08:00
recreate-8-4	passed	00:12:51
scaling-8-4	passed	00:10:16
scheduled-backup-8-0	passed	00:13:17
scheduled-backup-8-4	passed	00:15:17
service-per-pod-8-4	passed	00:07:10
sidecars-8-4	passed	00:04:31
smart-update-8-4	passed	00:09:30
storage-8-4	passed	00:03:54
telemetry-8-4	passed	00:05:59
tls-cert-manager-8-4	passed	00:10:03
users-8-0	passed	00:08:07
users-8-4	passed	00:07:53
version-service-8-4	passed	00:20:49

Summary	Value
Tests Run	59/59
Job Duration	01:56:52
Total Test Time	11:02:35

commit: https://github.com/percona/percona-server-mysql-operator/pull/1149/commits/9302e4b51980585bb3f8e95801ec867aacc50db9 image: perconalab/percona-server-mysql-operator:PR-1149-9302e4b5