Nextcloud Upgrade purges mounted external local storage via rsync

[mirisbowring]

trafficstars

Hi, I've reported this issue already but it got removed to the docker repository https://github.com/nextcloud/docker/issues/1493 (which is false in this case). See details below!

Steps to reproduce

1. Vanilla Nextcloud instance
2. mount external local storage to nextcloud instance
3. add files
4. upgrade nextcloud to next version when available

Now, when starting the application, rsync delete gets called on EVERY file on my external local storage. Due to this Bug I lost 2 Times about 16 TB of Data. (Upgrade to 21.0.1 and to 21.0.2). After the second time i mounted the storage as read only. Rsync now throws the error that it cant delete the file. Therefore Nextcloud wont start.

WHY does nextcloud try to delete the external storage???

Expected behaviour

The files should NOT be touched or even DELETED!

Actual behaviour

Nextcloud tries to wipe the whole external storage!

Server configuration

Operating system: Ubuntu 20.04 (Docker-Compose - default config)

Web server: Apache

Database: MariaDB

PHP version: 7.4.21

Nextcloud version: 21.0.3

Updated from an older Nextcloud/ownCloud or fresh install: from 21.0.2 / 21.0.1

Where did you install Nextcloud from: official docker repo

Signing status: cannot login because app won't start due to failing rsync delete

List of activated apps:

App list

Enabled:
 - accessibility: 1.7.0
 - activity: 2.14.3
 - breezedark: 21.0.8
 - bruteforcesettings: 2.2.0
 - checksum: 1.1.2
 - cloud_federation_api: 1.4.0
 - comments: 1.11.0
 - contactsinteraction: 1.2.0
 - dav: 1.17.1
 - extract: 1.3.1
 - federatedfilesharing: 1.11.0
 - federation: 1.11.0
 - files: 1.16.0
  - files_downloadactivity: 1.10.0
  - files_external: 1.12.0
  - files_pdfviewer: 2.1.0
  - files_rightclick: 1.0.0
  - files_sharing: 1.13.1
  - files_trackdownloads: 1.10.0
  - files_trashbin: 1.11.0
  - files_versions: 1.14.0
  - files_videoplayer: 1.10.0
  - firstrunwizard: 2.10.0
  - group_everyone: 0.1.7
  - groupfolders: 9.0.2
  - logreader: 2.6.0
  - lookup_server_connector: 1.9.0
  - metadata: 0.13.0
  - notifications: 2.9.0
  - oauth2: 1.9.0
  - password_policy: 1.11.0
  - provisioning_api: 1.11.0
  - serverinfo: 1.11.0
  - settings: 1.3.0
  - sharebymail: 1.11.0
  - systemtags: 1.11.0
  - text: 3.2.0
  - theming: 1.12.0
  - twofactor_backupcodes: 1.10.0
  - twofactor_webauthn: 0.2.10
  - updatenotification: 1.11.0
  - viewer: 1.5.0
  - workflowengine: 2.3.0
Disabled:
  - admin_audit
  - dashboard
  - encryption
  - nextcloud_announcements
  - photos
  - previewgenerator
  - privacy
  - recommendations
  - support
  - survey_client
  - user_ldap
  - user_status
  - weather_status

Nextcloud configuration:

Config report

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "nextcloud.primboard.de",
        "overwriteprotocol": "https",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.homelab-01.primboard.de"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "21.0.2.1",
        "overwrite.cli.url": "https:\/\/nextcloud.homelab-01.primboard.de",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "PLAIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "loglevel": 2,
        "maintenance": false
    }
}

Are you using external storage, if yes which one: local

Are you using encryption: no

Are you using an external user-backend, if yes which one: NO

Client configuration

Browser: Chrome

Operating system: MacOS

Logs

Configuring Redis as session handler
Initializing nextcloud 21.0.3.1 ...
Upgrading nextcloud from 21.0.2.1 ...
cannot delete non-empty directory: moviedb1/Movies/Movie 1/Sample
cannot delete non-empty directory: moviedb1/Movies/Movie 1
cannot delete non-empty directory: moviedb1/Movies/Movie 2/Sample
cannot delete non-empty directory: moviedb1/Movies/Movie 2
cannot delete non-empty directory: moviedb1/Movies/Movie 3/Sample
cannot delete non-empty directory: moviedb1/Movies/Movie 3
cannot delete non-empty directory: moviedb1/Movies/Movie 4
cannot delete non-empty directory: moviedb1/Movies/Movie 5
cannot delete non-empty directory: moviedb1/Movies/Movie 6
cannot delete non-empty directory: moviedb1/Movies/Movie 7/Sample
cannot delete non-empty directory: moviedb1/Movies/Movie 7
cannot delete non-empty directory: moviedb1/Movies/Movie 8
cannot delete non-empty directory: moviedb1/Movies/Movie 9
cannot delete non-empty directory: moviedb1/Movies/Movie 10/Sample
cannot delete non-empty directory: moviedb1/Movies/Movie 10
cannot delete non-empty directory: moviedb1/Movies
cannot delete non-empty directory: moviedb1/download/Movie100/Sample
cannot delete non-empty directory: moviedb1/download/Movie100
cannot delete non-empty directory: moviedb1/download
cannot delete non-empty directory: moviedb1/Shows/Show A/Episode-1.h264.mkv
cannot delete non-empty directory: moviedb1/Shows/Show A/Episode-2.h264.mkv
cannot delete non-empty directory: moviedb1/Shows/Show A/Episode-3.h264.mkv
cannot delete non-empty directory: moviedb1/Shows/Show A/Episode-4.h264.mkv
cannot delete non-empty directory: moviedb1/Shows/Show A/Episode-5.h264.mkv
cannot delete non-empty directory: moviedb1/Shows/Show A/Episode-6.h264.mkv
cannot delete non-empty directory: moviedb1/Shows/Show A/Episode-7.h264.mkv
...

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]

Web server error log

The above is the docker output log

Nextcloud log (data/nextcloud.log)

Nextcloud log

This logfile was empty

Browser log

Browser log

cannot access nextcloud because it's retarting due to the enforced error

[JuniorJPDJ]

Where is your external storage mounted?

[mirisbowring]

There are two mounts. One ist exfat formatted and mounted as read-write with user/group www-data:root to /media/mount1/

The second is ext4 formatted and mounted to /media/mount2

[JuniorJPDJ]

I'm asking about paths inside docker. That's it?

[mirisbowring]

Ah sry, misunderstood that.

they are mounted to /var/www/html/disk1 oder disk2 inside the container

[JuniorJPDJ]

Then well... It's your mistake. Maybe containers behavior is not documented good enough, but it removes everything in /var/www/html/ to upgrade installed nextcloud clearly. There are 4 exceptions: config data custom_apps themes Put your directories out of /var/www/html/ or inside one of mentioned directories and everything will be ok ;)

Aaaand.. Next time just don't trust maintainers, read scripts you are installing ;p https://github.com/nextcloud/docker/blob/master/docker-entrypoint.sh#L104

[mirisbowring]

Yeah I saw the line, but I wondered if this is really the expected behavior. Thank you for clarification!

Will this be added to documentation?

[JuniorJPDJ]

It should be, but someone would need to do this I'm not maintainer, I'm just an user ;D

[mabeett]

Hello, Niether am I maintainer but I'd suggest you to propose changes in README.md under the tittle Persistent Data. Maybe a more explicit text results helpful people as @mirisbowring in the past. When creating the PR please mention this issue, so the mantainer could get the context of the intervention.

[ChristianKniep]

On that one, I am wondering why do you even need to rsync the installation onto the persistent storage? Another way of doing that would be to just augment the persistent paths. Like within the container:

1. move /usr/src/nextcloud/ to /var/www/html
2. have volumes mounted as /var/www/html/{config, data, custom_apps, themes} from underneath
3. if those volumes are empty, seed them with a default (like you do in the entrypoint)
4. ideally make the container file-system read-only and create volumes for everything that should be r/w.

With the work that @f7o does in aws-samples/aws-serverless-nextcloud volumes are used for some of them, expect themes that is. 🤔

As a docker nutjob, I am wondering whether the persistance of /var/www/html as a whole is necessary?

[ChristianKniep]

I created a PR to illustrate the point #1548

[stsecurity]

Just experienced the same thing and found this disscussion, I think at least some changes should be made to the README.md as it clearly states in the 'Update to a newer version' section:

Since all data is stored in volumes, nothing gets lost.

It is natural to assume all the data under /var/www/html would be preseved as we have mounted that directory as volume.

[J0WI]

Note added in https://github.com/nextcloud/docker/pull/1862

[9-FS]

Then well... It's your mistake. Maybe containers behavior is not documented good enough, but it removes everything in /var/www/html/ to upgrade installed nextcloud clearly. There are 4 exceptions: config data custom_apps themes Put your directories out of /var/www/html/ or inside one of mentioned directories and everything will be ok ;)

Aaaand.. Next time just don't trust maintainers, read scripts you are installing ;p https://github.com/nextcloud/docker/blob/master/docker-entrypoint.sh#L104

I wish I would have known this earlier. I have just lost my whole media collection.