helm deployed nextcloud make CNPG deployed postgres stop due to low disk space

Open zeddit opened this issue 11 months ago • 0 comments

Describe your Issue

I am using helm to deploy a nextcloud app but sometime it fails and can not restart normally. after some digs, I found that it's because the database, i.e. postgres failed. the postgres failed due to an error of low disk space which means it consumed all the available space in pvc. However, the space is as large as 100GB, I think it's quite large enough.

when I expand the disk, the pg and nextcloud will recovery and I checked the recovered postgres, it used less than 1GB data. what's more, after 1-2 weeks, the same thing occurred again, I have to expand the storage again. and 2 weeks later, the same thing occurred again.

I am using the project of cnpg to deploy a postgres of 3 intances. https://cloudnative-pg.io/

Logs and Errors

{"level":"info","ts":"2025-01-01T04:02:02Z","logger":"setup","msg":"Checking for free disk space for WALs before starting PostgreSQL","logging_pod":"nextcloud-pg-1"}
{"level":"info","ts":"2025-01-01T04:02:02Z","logger":"setup","msg":"Detected low-disk space condition, avoid starting the instance","logging_pod":"nextcloud-pg-1"}

Describe your Environment

Kubernetes distribution: RKE2.
Helm Version (or App that manages helm): version.BuildInfo{Version:"v3.16.1", GitCommit:"5a5449dc42be07001fd5771d56429132984ab3ab", GitTreeState:"clean", GoVersion:"go1.22.7"}
Helm Chart Version: nextcloud nextcloud 6 2024-11-30 20:18:02.321695285 +0800 CST deployed nextcloud-6.2.4 30.0.2
values.yaml:

## Official nextcloud image version
## ref: https://hub.docker.com/r/library/nextcloud/tags/
##
image:
  repository: nextcloud
  flavor: apache
  # default is generated by flavor and appVersion
  tag: 30.0.2-apache
  pullPolicy: IfNotPresent
  # pullSecrets:
  #   - myRegistrKeySecretName
 
# Number of replicas to be deployed
replicaCount: 1
ingress:
  enabled: true
  className: nginx
  annotations: 
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/cors-allow-headers: X-Forwarded-For
    nginx.ingress.kubernetes.io/enable-cors: 'true'
    # Keep this in sync with the README.md:
    nginx.ingress.kubernetes.io/server-snippet: |-
      server_tokens off;
      proxy_hide_header X-Powered-By;
      rewrite ^/.well-known/webfinger /index.php/.well-known/webfinger last;
      rewrite ^/.well-known/nodeinfo /index.php/.well-known/nodeinfo last;
      rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
      rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;
      location = /.well-known/carddav {
        return 301 $scheme://$host/remote.php/dav;
      }
      location = /.well-known/caldav {
        return 301 $scheme://$host/remote.php/dav;
      }
      location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
      }
      location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
        deny all;
      }
      location ~ ^/(?:autotest|occ|issue|indie|db_|console) {
        deny all;
      }
  tls:
    - secretName: nextcloud-cert
      hosts:
        - pan.xxx.tech
  labels: {}
  path: /
  pathType: Prefix

# Allow configuration of lifecycle hooks
# ref: https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/
lifecycle: {}
  # postStartCommand: []
  # preStopCommand: []
  
nextcloud:
  host: pan.xxx.tech
  username: admin
  password: xxx@Nc
  ## Use an existing secret
  existingSecret:
    enabled: false
  update: 0
  # If web server is not binding default port, you can define it
  containerPort: 80
  datadir: /var/www/html/data
  # if set, we'll template this list to the NEXTCLOUD_TRUSTED_DOMAINS env var
  trustedDomains: [] # forbidden access from ip if not set here!
  ## SMTP configuration
  mail: 
    enabled: false
    # the user we send email as
    fromAddress: admin
    # the domain we send email from
    domain: xxx.tech
    smtp:
      host: domain.com
      secure: ssl
      port: 465
      authtype: LOGIN
      name: user
      password: pass

  ## Primary ObjectStore options
  # see: https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html#configuring-object-storage-as-primary-storage
  objectStore:
    # https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html#simple-storage-service-s3
    s3:
      enabled: true
      accessKey: "xxxx"
      secretKey: "xxxx"
      host: "xxx.xxxx.xxx"
      ssl: true
      port: "443"
      region: "us-east-1" # minio default
      bucket: "nextcloud"
      prefix: "" # object prefix in bucket
      usePathStyle: true
      autoCreate: false # autocreate the bucket
      storageClass: "STANDARD"

  # Extra config files created in /var/www/html/config/
  # ref: https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html#multiple-config-php-file
  configs:
    proxy.config.php: |-
      <?php
      $CONFIG = array (
        'trusted_proxies' => array(
          0 => '127.0.0.1',
          1 => '10.0.0.0/8',
        ),
        'forwarded_for_headers' => array('HTTP_X_FORWARDED_FOR'),
      );
    appstore.config.php: |-
      <?php
      $CONFIG = array (
        'appstoreenabled' => false,
      );
    oidc.config.php: |-
      <?php
      $CONFIG = array (
        'allow_local_remote_servers' => true,
      );

  extraEnv:
    - name: "TZ"
      value: "Asia/Shanghai"

  # Extra mounts for the pods. Example shown is for connecting a legacy NFS volume
  # to NextCloud pods in Kubernetes. This can then be configured in External Storage
  extraVolumes:
  #  - name: nfs
  #    nfs:
  #      server: "10.0.0.1"
  #      path: "/nextcloud_data"
  #      readOnly: false
  extraVolumeMounts:
  #  - name: nfs
  #    mountPath: "/legacy_data"
 
  # Set securityContext parameters for the nextcloud CONTAINER only (will not affect nginx container).
  # For example, you may need to define runAsNonRoot directive
  securityContext: 
    runAsUser: 33
    runAsGroup: 33
    runAsNonRoot: true
    readOnlyRootFilesystem: false

  # Set securityContext parameters for the entire pod. For example, you may need to define runAsNonRoot directive
  podSecurityContext: {}
  
internalDatabase:
  enabled: false
  name: nextcloud

##
## External database configuration
##
externalDatabase:
  enabled: true
  type: postgresql
  host: "nextcloud-pg-rw:5432"
  database: nextcloud
  existingSecret:
    enabled: true
    secretName: nextclouduser-secret
    usernameKey: username
    passwordKey: password

mariadb: 
  enabled: false # we use postgres

redis:
  enabled: false

## Cronjob to execute Nextcloud background tasks
## ref: https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/background_jobs_configuration.html#cron
##
cronjob:
  enabled: false

service:
  type: ClusterIP
  port: 8080

## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  # Nextcloud Data (/var/www/html)
  enabled: true
  annotations: {}
  storageClass: ""
  accessMode: ReadWriteOnce
  size: 8Gi
  ## Use an additional pvc for the data directory rather than a subpath of the default PVC
  ## Useful to store data on a different storageClass (e.g. on slower disks)
  nextcloudData:
    enabled: false # data part, means the whole is not only data

resources: 
  requests:
    memory: "8Gi"
    cpu: "4"

## Enable pod autoscaling using HorizontalPodAutoscaler
## ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
##
hpa:
  enabled: false
  cputhreshold: 60
  minPods: 1
  maxPods: 10

nodeSelector: {}

## Prometheus Exporter / Metrics
##
metrics:
  enabled: false

rbac:
  enabled: false
  serviceaccount:
    create: true
    name: nextcloud-serviceaccount
    annotations: {}

Jan 01 '25 06:01 zeddit