cms
cms copied to clipboard
craftcms
[3.x]: Files are lost (deleted) after moving a folder in Assets
What happened?
Description
We have Assets connected to an S3 bucket. When we move a folder with a large number of files (approx 2000), not all the files are relocated. Instead, some "go missing" (i.e. are deleted) once the process is complete. This does not happen when the folder has a smaller number of files (i.e. 10).
Steps to reproduce
- Start with a folder which contains 2000 files
- Move the folder somewhere else (i.e. into another folder)
- Look at how many files are left; it should be less than 2000
Expected behavior
No files are lost after moving the folder.
Actual behavior
Files are lost (i.e deleted) after moving the folder.
Craft CMS version
3.8.16
PHP version
8.0.30
Operating system and version
Linux 4.4.0-1128-aws
Database type and version
MySQL 8.0.28
Image driver and version
GD 8.0.30
Installed plugins and versions
Amazon S3 1.3.2 Asset Usage 2.3.1 Chunked File Uploads 1.1.1 CP Field Inspect 1.4.4 Embedded Assets 2.11.4 Environment Label 3.2.0 Feed Me 4.7.0 ImageOptimize 1.6.51 Queue Manager 1.2.0 Redactor 2.10.12 Restrict Asset Delete 2.0.0 Retcon 2.7.1
Unfortunately, we made the same experience a long time ago. Three times we had to use a self-written script to restore the files from a backup bucket, to which we always automatically copy all objects per livecycle rule and which has versioning on. With a database dump shortly before the incident.
This is the reason why we actually don't allow any user to delete a file anymore, because this is the only protection against this problem. Especially the moving of a folder(-structure) by drag'n'drop without prompting in version 3.x is a source for unintentional moving and the unpleasant result described here.
If I remember correctly, the root cause was (at least with us) that the move is performed synchronously with the request and not asynchronously per job in the background (queue) and the move within an object store takes much longer than on a local file system. The Ajax request is aborted after 60 seconds by Cloudfront and then the request continues to run for a while on the server until it also runs into the max execution time there. But I can't remember exactly, I would have to look into the codebase of 3.x again.
@kringkaste thanks for sharing your experience. We're considering running some tests where deleting files is restricted.
It seems this can be done at either the AWS level (via a policy) or just be restricting user permissions within Craft. I couldn't tell from your response which you used. Applying it to AWS seems the safest but I'm concerned this may mess with how Assets behaves... i.e. if a file is moved, will could see it replicated in different folders.
We restricting user permissions in Craft. It's better to do this on application level.