Nextcloud pods failed with EFS as persistent volume

[SerhiiK]

Describe your Issue

Nextcldoud can't work properly with EFS as persistent volume through user and group issues.

Logs and Errors

If use default storageClass faced permission issue

Logs

Initializing nextcloud 27.0.2.1 ...
rsync: [generator] chown "/var/www/html/." failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/.github" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/.github/workflows" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-crt-php" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-crt-php/ext" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-crt-php/src" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-crt-php/src/AWS" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-crt-php/src/AWS/CRT" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-crt-php/src/AWS/CRT/Auth" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-crt-php/src/AWS/CRT/HTTP" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-crt-php/src/AWS/CRT/IO" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-crt-php/src/AWS/CRT/Internal" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ACMPCA" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/EMRServerless" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/EMRServerless/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Ec2" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Ec2/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Ecr" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Ecr/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Ecs" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Ecs/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Efs" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Efs/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElastiCache" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElastiCache/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticBeanstalk" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticBeanstalk/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticInference" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticInference/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticLoadBalancing" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticLoadBalancing/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticLoadBalancingV2" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticLoadBalancingV2/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticTranscoder" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticTranscoder/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticsearchService" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/ElasticsearchService/Exception" failed: Operation not permitted (1)
....
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Emr" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Emr/Exception" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Endpoint" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Endpoint/UseDualstackEndpoint" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Endpoint/UseDualstackEndpoint/Exception" failed: Operation not permitted (1)
 rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Endpoint/UseFipsEndpoint" failed: Operation not permitted (1)
rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/Endpoint/UseFipsEndpoint/Exception" failed: Operation not permitted (1)
 rsync: [generator] chown "/var/www/html/3rdparty/aws/aws-sdk-php/src/EndpointDiscovery" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.htaccess.nDcRHg" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.user.ini.7Bw0B5" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.AUTHORS.N71yXb" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.COPYING.v9zX8I" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.console.php.WnbczZ" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.cron.php.jsVxby" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.index.html.PS802l" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.index.php.NDpOn3" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.occ.R7KEHH" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.public.php.3RP2WP" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.remote.php.FKkukp" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.robots.txt.d62oBz" failed: Operation not permitted (1)
rsync: [receiver] chown "/var/www/html/.status.php.6Xqguy" failed: Operation not permitted (1)

Efs driver by default can use random POSIX user id and group id. In this example is 1004:1004. Try to fix it by creating special storageClass with parameters uid and gid

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-nextcloud
mountOptions:
  - tls
parameters:
  basePath: /dynamic_provisioning
  directoryPerms: '700'
  fileSystemId: fs-[ID]
  gid: '33'
  provisioningMode: efs-ap
  uid: '33'
provisioner: efs.csi.aws.com
reclaimPolicy: Retain
volumeBindingMode: Immediate

After it all files has owner www-data:www-data but nextcloud shows next logs:

Logs

crond: crond (busybox 1.30.1) started, log level 0
crond: user:www-data entry:(null)
100001000010000100001000010000100001000010000100001000010000
111111111111111111111111
11111111111111111111111111111111
111111111111
1111111
crond: user:www-data entry:(null)
100001000010000100001000010000100001000010000100001000010000
111111111111111111111111
11111111111111111111111111111111
111111111111
1111111
crond: wakeup dt=48
crond: file www-data:
crond:  line php -f /var/www/html/cron.php
crond:  job: 0 php -f /var/www/html/cron.php
crond: child running /bin/bash
crond: USER www-data pid   7 cmd php -f /var/www/html/cron.php

Warning: require_once(/var/www/html/lib/versioncheck.php): Failed to open stream: No such file or directory in /var/www/html/cron.php on line 40

Fatal error: Uncaught Error: Failed opening required '/var/www/html/lib/versioncheck.php' (include_path='.:/usr/local/lib/php') in /var/www/html/cron.php:40
Stack trace:
#0 {main}
  thrown in /var/www/html/cron.php on line 40
crond: wakeup dt=10
crond: wakeup dt=50
crond: file www-data:
crond:  line php -f /var/www/html/cron.php

Nextcloud container in pod can't pass probes and failed.

Describe your Environment

• Kubernetes distribution: EKS 1.23.17

• Use efs csi driver with dynamic provisioning

• Helm Chart Version: ~~3.5.1~~ 4.2.0

• image: "nextcloud:27.0.2-apache"

• values.yaml:

        internalDatabase:
          enabled: false

        externalDatabase:
          enabled: true
          host: [URL]
          database: next
          user: admin
          password: pass

        persistence:
          enabled: true
          storageClass: "efs-nextcloud"
          accessMode: ReadWriteMany

        cronjob:
          enabled: true

        nextcloud:
          host: [HOST]
          podSecurityContext:
            fsGroup: 33
          defaultConfigs:
            autoconfig.php: true
          configs:
            test.config.php: |-
              <?php
              $CONFIG = array (
                'instanceid' => getenv('CONFIG_INSTANCEID'),
                'overwrite.cli.url' => '[HOST]',
                'passwordsalt' => getenv('CONFIG_PASSWORDSALT'),
                'secret' => getenv('CONFIG_SECRET'),
                'trusted_domains' => 
                  array (
                    0 => '[HOST]',
                  ),
                'overwrite.cli.url' => '[HOST]',
                'overwriteprotocol' => 'https',  
                'objectstore' => 
                  array (
                    'class' => '\\OC\\Files\\ObjectStore\\S3',
                    'arguments' => 
                      array (
                        'bucket' => getenv('CONFIG_BUCKET'),
                        'autocreate' => true,
                        'key' => getenv('CONFIG_S3_ACCESS_KEY'),
                        'secret' => getenv('CONFIG_S3_SECRET_KEY'),
                      ),
                  ),
              );

          extraEnv:
          ...

        ingress:     
          enabled: true
          ...

[jessebot]

Thanks for submitting an issue!

In the future, please post all logs as text instead of an image, so that other users can search your errors in the future, and to make this more accessible to users with vision disabilities as since there's no alt text, a screen reader app used by someone who is blind or has no vision, will just show that as "image" with no further context.

If this is not your complete values.yaml, could you please post your whole values.yaml? This helps us evaluate all possible issues better. For instance, are you using nginx? which nextcloud image flavor/tag are you using? Depending on these answers, the security context user changes. Some directories are always owned by root as well, so that's something to keep in mind.

Can you also try this with the latest version of the nextcloud helm chart (4.1.0) and let us know if the issue persists? There's been some minor changes as well as the nextcloud image itself that could be at play and we just want to make sure.

[jessebot]

I think #318 may also be the same issue.