Impossible to move an instance on S3/SWIFT while using GroupFolders

[solracsf]

We have an instance using S3 as primary storage. We we're planning to move it from one server to another, problem is that after moving it (different server, different providers for both servers and S3), any GroupFolders folder is...EMPTY.

Problem is inexistant if there is no GroupFolders involved, using the migration plan above.

Step of migration

While in maintenance mode:

1. rsync all /public folder contents from one server to another
2. mysqldump --single-transaction nextcloud > nextcloud.sql and import on the new server
3. Adjust config.php to use new credentials
4. Using rclone or mc (tried with both), sync all content from one S3 provider to another. At the end, the two S3 buckets match perfectly (number of objects, size, and each object matches it's MD5).
5. Switch OFF maintenance mode on new server, and log in.

So at the end:

• same files (code base)
• same contents on S3 buckets
• same SQL base (same MySQL version on old and new servers)
• all operations done while NC was in maintenance mode

Power-up new instance by switching DNS entries (hosts file in my case), every group folder is EMPTY on the new instance. All folders are there, but empty. Everything that is NOT on a GroupFolder, is completely fine (files, shares, activities, users, calendars...). So the problem is somewhere with GroupFolder not detecting its contents on the new S3 bucket...

Also, every GroupFolder has a modified date as "Few seconds ago" just after log in to the new instance, which of course is not true. But maybe this is a lead to find the root cause.

We've tried this procedure moving from S3 > S3 or from S3 > SWIFT, the result is the same. We've switched many Nextcloud instances not using GroupFolders, using the above procedure, no problems.

Folder(s) scan doesn't help either:

php occ groupfolders:scan 1
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 1       | 0     | 00:00:00     |
+---------+-------+--------------+

[chapa]

Hello, I just came across this issue by chance, not sure if this is still relevant but maybe I can help those who will end up here and need a solution.

The same thing happened to me some weeks ago when I tried to migrate from one S3 storage to another (same application server, same S3 provider, but different bucket names).

I found that Nextcloud generates a new entry in the oc_storages table when the objectstore configuration is changed:

+------------+--------------------------------+-----------+--------------+
| numeric_id | id                             | available | last_checked |
+------------+--------------------------------+-----------+--------------+
|          1 | object::store:amazon::bucket-1 |         1 |         NULL |
+------------+--------------------------------+-----------+--------------+

Becomes:

+------------+--------------------------------+-----------+--------------+
| numeric_id | id                             | available | last_checked |
+------------+--------------------------------+-----------+--------------+
|          1 | object::store:amazon::bucket-1 |         1 |         NULL |
|          2 | object::store:amazon::bucket-2 |         1 |         NULL |
+------------+--------------------------------+-----------+--------------+

The problem is that files in group folders are "stored" in those user-independent storages:

+--------+---------+--------------------+
| fileid | storage | path               |
+--------+---------+--------------------+
|      1 |       1 | __groupfolders/... |
+--------+---------+--------------------+

Changing the name of the bucket causes the change of the "current" storage (1 becomes 2), so groupfolders looks for files in 2 but all files still are in storage 1.

You have to keep the same numeric_id for the "current" storage, I did it by removing the additional storage and renaming the id of the old storage, something like:

DELETE oc_storages WHERE id = 'object::store:amazon::bucket-2';
UPDATE oc_storages SET id = 'object::store:amazon::bucket-2' WHERE id = 'object::store:amazon::bucket-1';

[provokateurin]

We still need some proper documentation or even better a way to avoid this problem all together.

[provokateurin]

Could this even be a duplicate of https://github.com/nextcloud/groupfolders/issues/1468 as it also seems to happen on other storage backends?

[raboof]

I think I encountered this issue while migrating a Nextcloud instance according to https://docs.nextcloud.com/server/latest/admin_manual/maintenance/migrating.html , where the storage folder on the new machine was in a different location compared to the old machine. Indeed the oc_storages folder had the old storage at numeric id '1' and the new storage at a new storage location id

I seem to have worked around it by setting update oc_storages set available = 0 where numeric_id = 1, though I'm not familiar enough with Nextcloud yet to confidently say this is a proper solution :) .