Distribution Redis container shuts down every minute

[dennisheller-nokia]

Is this a request for help?: no

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes: Helm: v3.8.1 Kubernetes: v1.22.9

Which chart: distribution

What happened: Redis container did shut down every minute without error message. This caused the whole Distribution service to be unusable. Restart count of the pod increased to a number of thousands.

What you expected to happen: Redis container keeps running so that Distribution can do its work.

How to reproduce it (as minimally and precisely as possible): Install Helm chart jfrog/distribution to a Kubernetes cluster and trigger a few distributions. As soon as Redis DB increases in size, the propability of the bug increases.

Anything else we need to know: I found out that the shutdown is initiated by the Kubernetes cluster because the liveliness probes fail:

livenessProbe:
  exec:
    command:
      - redis-cli
      - ping
  initialDelaySeconds: 30
  timeoutSeconds: 1
  periodSeconds: 10
  successThreshold: 1
  failureThreshold: 3
readinessProbe:
  exec:
    command:
      - redis-cli
      - ping
  initialDelaySeconds: 10
  timeoutSeconds: 1
  periodSeconds: 10
  successThreshold: 1
  failureThreshold: 3

The timeout of 1 second is too low. The fact that the container is shut down after 1 minute also proves that this is the cause, because initialDelySeconds + (failureThreshold x periodSeconds) = 30s + (3s x 10s) = 60s

[chukka]

Update : This issue will be resolved along with distribution 2.14.x release

[chukka]

Fixed via #1656