charts icon indicating copy to clipboard operation
charts copied to clipboard

Can't open /bitnami/redis/data/nodes.conf in order to acquire a lock: Permission denied in redis-cluster helm chart

Open abhishekGupta2205 opened this issue 3 years ago • 8 comments

Name and Version

bitnami/redis-cluster , 7.1.0

What steps will reproduce the bug?

  1. helm install redis-cluster redis-cluster
  2. in values.yaml change
podSecurityContext:
  enabled: true
  fsGroup: 0
  runAsUser: 0
containerSecurityContext:
  enabled: true
  runAsUser: 0
  runAsNonRoot: false
  1. now giving file path "var/log/redis.log" in logfile parameter in configmap part .

Are you using any custom parameters or values?

No response

What is the expected behavior?

log should be stored in var/log/redis.log

What do you see instead?

crashloopbackoff occurs

Additional information

Can't open /bitnami/redis/data/nodes.conf in order to acquire a lock: Permission denied

abhishekGupta2205 avatar Jul 05 '22 12:07 abhishekGupta2205

Hi @abhishekgupta2205

The real reason behind this is that the redis process does not have enough permissions to write on var/log/redis.log. By default, even if the container runs as a root, the redis process will run as the unprivileged user redis:

$ kubectl logs test-redis-cluster-0
COPYING FILE
redis-cluster 12:25:10.19
redis-cluster 12:25:10.19 Welcome to the Bitnami redis-cluster container
redis-cluster 12:25:10.19 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis-cluster
redis-cluster 12:25:10.19 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis-cluster/issues
redis-cluster 12:25:10.20
redis-cluster 12:25:10.20 INFO  ==> ** Starting Redis setup **
redis-cluster 12:25:10.27 INFO  ==> Initializing Redis
redis-cluster 12:25:10.32 INFO  ==> Setting Redis config file
Storing map with hostnames and IPs
redis-cluster 12:25:40.73 INFO  ==> ** Redis setup finished! **


*** FATAL CONFIG FILE ERROR (Redis 6.2.7) ***
Reading the configuration file, at line 241
>>> 'logfile "/var/log/redis.log"'
Can't open the log file: Permission denied

$ kubectl exec -it lambda-redis-cluster-0 -- whoami
root

$ kubectl exec -it lambda-redis-cluster-0 -- ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
redis          1  0.3  0.1  58456  8308 ?        Ssl  14:13   0:13 redis-server
root       17531  0.0  0.0   6696  2972 pts/0    Rs+  15:14   0:00 ps aux

This is in line with Redis' code security recomentations.

You have two options here:

  • Use a path where redis has enough permissions to write, such as /opt/bitnami/redis/logs or /tmp.
  • Apply the correct permissions to your desired path using an initContainer. You may use the existing property redis.initContainers for the matter.

joancafom avatar Jul 06 '22 15:07 joancafom

after removing logfile path also , i am facing the same error

abhishekGupta2205 avatar Jul 07 '22 04:07 abhishekGupta2205

Sorry, but I can't reproduce it on my side:

$ cat /tmp/custom-values.yaml
podSecurityContext:
  enabled: true
  fsGroup: 0
  runAsUser: 0
containerSecurityContext:
  enabled: true
  runAsUser: 0
  runAsNonRoot: false

$ helm install test bitnami/redis-cluster -f /tmp/custom-values.yaml
NAME: test
LAST DEPLOYED: Mon Jul 11 19:32:16 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis-cluster
CHART VERSION: 7.6.4
APP VERSION: 6.2.7** Please be patient while the chart is being deployed **


To get your password run:
    export REDIS_PASSWORD=$(kubectl get secret --namespace "default" test-redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d)

You have deployed a Redis® Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis® cluster:

1. Run a Redis® pod that you can use as a client:
kubectl run --namespace default test-redis-cluster-client --rm --tty -i --restart='Never' \
 --env REDIS_PASSWORD=$REDIS_PASSWORD \
--image docker.io/bitnami/redis-cluster:6.2.7-debian-11-r9 -- bash

2. Connect using the Redis® CLI:

redis-cli -c -h test-redis-cluster -a $REDIS_PASSWORD

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
test-redis-cluster-0   1/1     Running   0          58s
test-redis-cluster-1   1/1     Running   0          58s
test-redis-cluster-2   1/1     Running   0          58s
test-redis-cluster-3   1/1     Running   0          58s
test-redis-cluster-4   1/1     Running   0          58s
test-redis-cluster-5   1/1     Running   0          58s

Are you using the same custom values as me?

joancafom avatar Jul 11 '22 17:07 joancafom

Actually i want to change values of datadir nad logfile path of redis-cluster pods . so i tried changing them in configmap.yaml file present in template folder . so after running the helm chart like you did , i tried setting those values and then tried upgrading the chart . But this error occurs when i upgrade it .

i want to set my logfile path - /var/log/redis.log datadir - /var/lib/redis

abhishekGupta2205 avatar Jul 12 '22 06:07 abhishekGupta2205

As said in my previous comment, redis cannot write in those directories by default. You have two options:

  • Use a path where redis has enough permissions to write, such as /opt/bitnami/redis/logs or /tmp.
  • Apply the correct permissions to your desired path using an initContainer. You may use the existing property redis.initContainers for the matter.

joancafom avatar Jul 14 '22 09:07 joancafom

 initContainers: 
    - name: your-image-name
      image: busybox
      imagePullPolicy: Always
      command: ['sh', '-c', 'echo "hello world"']

i ran a simple echo command using init containers by defining it in values.yaml. ON upgrading the chart it shows same error. Here i am not changing any directory or something. Screenshot from 2022-07-18 18-19-20

abhishekGupta2205 avatar Jul 18 '22 12:07 abhishekGupta2205

ON upgrading the chart it shows same error

Are you upgrading from an older version?

On my side, I still cannot reproduce your issue. In order to do so:

  1. I have performed an installation using your provided values:
podSecurityContext:
  enabled: true
  fsGroup: 0
  runAsUser: 0
containerSecurityContext:
  enabled: true
  runAsUser: 0
  runAsNonRoot: false

This leads to a running Redis cluster (it has only restarted on start-up):

$ kubectl get pods
NAME                    READY   STATUS    RESTARTS      AGE
alpha-redis-cluster-0   1/1     Running   1 (22s ago)   88s
alpha-redis-cluster-1   1/1     Running   1 (21s ago)   88s
alpha-redis-cluster-2   1/1     Running   0             88s
alpha-redis-cluster-3   1/1     Running   1 (21s ago)   88s
alpha-redis-cluster-4   1/1     Running   1 (17s ago)   88s
alpha-redis-cluster-5   1/1     Running   0             88s
  1. Then, I perform an upgrade operation in which I specify the same initContainers as you:
$ helm upgrade alpha bitnami/redis-cluster --set password=$REDIS_PASSWORD
...
$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
alpha-redis-cluster-0   1/1     Running   0          18m
alpha-redis-cluster-1   1/1     Running   0          18m
alpha-redis-cluster-2   1/1     Running   0          18m
alpha-redis-cluster-3   1/1     Running   0          18m
alpha-redis-cluster-4   1/1     Running   0          19m
alpha-redis-cluster-5   1/1     Running   0          19m
$ kubectl logs alpha-redis-cluster-0 -c your-image-name
hello world

As you can see again, the cluster is running with no problems.

If something does not match what you are doing, could you please provide the exact steps and values you are using so that I can reproduce it on my side?

joancafom avatar Jul 25 '22 10:07 joancafom

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] avatar Aug 10 '22 01:08 github-actions[bot]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

github-actions[bot] avatar Aug 15 '22 01:08 github-actions[bot]