nextcloud
Feature: maintanence mode cron jobs
Description of the change
Add optional nextcloud kubernetes cronjobs with configurable schedules for going into and out of maintenance modes. Ideally the cronjobs just run the following for "maintenance on":
# indexes the current files into the database before entering maintenance mode
# we index current files so the database backups include their metadata
su -s /bin/sh www-data -c "php occ files:scan --all && php occ maintenance:mode --on"
And then for the "maintenance off" job we'd would run:
# just takes nextcloud of maintenance mode
su -s /bin/sh www-data -c "php occ maintenance:mode --off"
example cronjob for maintenance mode on
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: nextcloud-maintanence-mode-on
namespace: nextcloud
spec:
# daily at 2:00AM Central Europe Time
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
volumes:
- name: nextcloud-main
persistentVolumeClaim:
claimName: nextcloud-files
- configMap:
defaultMode: 420
name: nextcloud-web-app-config
name: nextcloud-config
- configMap:
defaultMode: 420
name: nextcloud-web-app-nginxconfig
name: nextcloud-nginx-config
# consider this to chown /var/www/html/data and /data to www-data:root
# initContainers:
# - name: nextcloud-maintanence-perm-fix
containers:
- name: nextcloud-maintanence
image: nextcloud:27.0.0-fpm
imagePullPolicy: IfNotPresent
command:
- /bin/bash
- -c
- su -s /bin/bash www-data -c "php occ files:scan --all && php occ maintenance:mode --on"
volumeMounts:
- mountPath: /var/www/
name: nextcloud-main
subPath: root
- mountPath: /var/www/html
name: nextcloud-main
subPath: html
# only needed if you have a seperate data dir for nextcloud
- mountPath: /var/www/html/data
name: nextcloud-main
subPath: data
- mountPath: /var/www/html/config
name: nextcloud-main
subPath: config
- mountPath: /var/www/html/custom_apps
name: nextcloud-main
subPath: custom_apps
- mountPath: /var/www/tmp
name: nextcloud-main
subPath: tmp
- mountPath: /var/www/html/themes
name: nextcloud-main
subPath: themes
- mountPath: /var/www/html/config/logging.config.php
name: nextcloud-config
subPath: logging.config.php
- mountPath: /var/www/html/config/proxy.config.php
name: nextcloud-config
subPath: proxy.config.php
- mountPath: /var/www/html/config/.htaccess
name: nextcloud-config
subPath: .htaccess
- mountPath: /var/www/html/config/apache-pretty-urls.config.php
name: nextcloud-config
subPath: apache-pretty-urls.config.php
- mountPath: /var/www/html/config/apcu.config.php
name: nextcloud-config
subPath: apcu.config.php
- mountPath: /var/www/html/config/apps.config.php
name: nextcloud-config
subPath: apps.config.php
- mountPath: /var/www/html/config/autoconfig.php
name: nextcloud-config
subPath: autoconfig.php
- mountPath: /var/www/html/config/smtp.config.php
name: nextcloud-config
subPath: smtp.config.php
env:
- name: POSTGRES_HOST
value: nextcloud-web-app-postgresql
- name: POSTGRES_DB
value: nextcloud
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
key: username
name: nextcloud-pgsql-credentials
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
key: nextcloudPassword
name: nextcloud-pgsql-credentials
- name: NEXTCLOUD_ADMIN_USER
valueFrom:
secretKeyRef:
key: username
name: nextcloud-admin-credentials
- name: NEXTCLOUD_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: nextcloud-admin-credentials
# - name: NEXTCLOUD_TRUSTED_DOMAINS
# value: cloud.example.com
- name: NEXTCLOUD_DATA_DIR
value: /var/www/html/data
Benefits
Right now, to create backups, you need to put nextcloud into maintenance mode and that would need to be done manually with an occ command. I'm using k8up to backup nextcloud, but it requires me to maintain two Kubernetes Cronjobs to put nextcloud into maintenance mode before the backup and then take it out of maintenance mode after the backup is done. The issue is that these cronjobs have to replicate the nextcloud pod, which means also keeping the same nextcloud image version, to avoid any issues with accidentally upgrading or downgrading anything.
If this is implemented, it would allow users to set a specific window that their backups could run through the use of something like k8up or velero or even just their own homegrown rsync or restic kubernetes cronjob.
Possible drawbacks
It's kind of confusing to have both our sidecar crobjob, for automated nextcloud maintanence tasks and this potentially additional kubernetes cronjob, however the Kubernetes side car job only handles background tasks, which can include backups, but the baked in nextcloud server backups are not done with restic, I don't think, and there's been complaints of the storage space it takes. I think this makes sense to implement anyway because it's one less manual k8s resource you need to maintain alongside the helm chart.
Additional information
I can submit a PR for this, but we'd have two cronjobs to template, and then the main values we'd be adding to values.yaml would be:
maintenance_mode_cronjobs:
# set to true to enable maintenance mode via a kubernetes cronjob during a specific window
enabled: false
# time to run the maintenance mode on job
# example to start maintenance mode daily at 2:00AM UTC
# start_schedule: "0 2 * * *"
start_schedule: ""
# time to run the maintenance mode off job
# example to stop maintenance mode daily at 3:00AM UTC
# end_schedule: "0 3 * * *"
end_schedule: ""
I'm personally going to be using s3 compliant object storage as my primary storage, but I still need maintenance mode for backing up the database.
ok, I re-read the nextcloud backup app's README and it looks like they do support differential backups, so I will test that, and if it works, I will report back and close this issue.
Update: Could not get the backups app to work, but it seems a lot of users have had issues with getting it to work in a container :shrug:
Have you considered pre and post hooks with your backup system? I've used velero's hooks successfully in the past for similar things.
velero was a bit too slow for me, so I switched to k8up. They have a concept of a pre-backup pod, but not a post-backup pod though.
My cronjobs are still hardcoded to be a specific nextcloud docker image tag, which is not great, because I have to remember to update them all the time, but unfortunately, that's just kind of the case until we would introduce a proper maintainence mode cronjob here. You can see my full setup here: https://github.com/small-hack/argocd-apps/tree/main/nextcloud
The cronjobs are here specifically: https://github.com/small-hack/argocd-apps/tree/main/nextcloud/maintenance_mode_cronjobs/templates