Consistent handling of last modified dates when copying/moving files

[despens]

Is your feature request related to a problem? Please describe.

Nextcloud handles last-modified metadata inconsistently when copying and moving files and folders:

Activity	Source	Destination	last modified preserved? v15	last modified preserved? v16	last modified preserved? v17	last modified preserved? v18	last modified preserved? v19
file upload > 10 MB via browser	Desktop	Primary Storage	✔️	✔️	✔️		✔️
file upload < 10 MB via browser	Desktop	Primary Storage	⛔	⛔	✔️		✔️
file upload via browser	Desktop	External Storage	⛔	⛔			⛔
sync files with Desktop client	Desktop	Primary Storage	✔️	✔️	✔️		✔️
upload files via WebDAV (@codiflow)	Desktop	Primary Storage	⛔		⛔	⛔	⛔
download files via WebDAV	Primary Storage	Desktop	✔️				⛔
upload files via rclone	Desktop	Primary Storage	✔️	✔️			✔️
upload files via rclone	Desktop	External Storage	⛔	⛔
moving files and folders within nextcloud	Primary Storage	Primary Storage	✔️	✔️	✔️		✔️
moving files and folders within nextcloud	Primary Storage	External Storage	⛔	⛔			⛔
moving files and folders within nextcloud	External Storage	Primary Storage	⛔	⛔			⛔
copy files and folders within nextcloud	Any Storage	Any Storage	⛔	⛔	⛔		⛔
copying files via cloud federation	remote nextcloud	local nextcloud	⛔				⛔
upload via file drop	Desktop	primary storage	⛔		⛔		⛔

(Primary storage refers to object storage directly managed by nextcloud, External Storage refers to externally mounted object storage.)

Describe the solution you'd like

Last-modification metadata should always be preserved, no matter where files and folders are coming from or moving to.

Describe alternatives you've considered

This should just work :wink:

Additional context

For my use case, the last modification date of files and folders is critical for comparing different versions of the same item at different locations. It also is important when moving files from active usage into an archival storage (nearline bucket mounted externally).

(related #15177, #14982)

[codiflow]

I can add that uploading files into nextcloud by using WebDAV also doesn't preserve the original timestamps.

[despens]

Thx @codiflow, added info to the table, please check if this is correct. Did you try with several WebDAV clients?

[codiflow]

Thx @codiflow, added info to the table, please check if this is correct. Did you try with several WebDAV clients?

I use Krusader on Manjaro Linux and Windows Explorer on Windows 8.1 Pro. With both the timestamp will be set to now on upload. It also doesn't matter if the file is smaller or larger than 10 MB.

[CherAlban]

I can confirm the <>10MB behavior with Firefox 66.0.3 (64-bit). Using WebDav in BeyondCompare the timestamp always gets set to now on upload. (which is very annoying for my use case)

[despens]

Using WebDAV via Gnome Shell, can confirm that last-modified dates are not preserved when moving items to nextcloud.

[despens]

Updated table again, found out that copying data from nextcloud to local computer via WebDAV preserves last modified date.

[codiflow]

I got a workaround for those who don't want to wait until this gets fixed:

1. Create a "sync" folder in Nextcloud
2. Install nextcloud client on your pc and sync this "sync" folder with to your pc
3. Move files whose timestamps should get preserved into this folder the nextcloud client is watching for changes on. Wait until your files got synced to the nextcloud server.
4. Go to the nextcloud WebUI and move the files from "sync" folder (with right timestamp) to the right destination

It's somehow complicated but prevents me from the need to sync the entire nextcloud with my PC.

[despens]

Tested with rclone and added to table.

[despens]

Tested with nextcloud 16.0.1 and added to table.

[despens]

Updated the table with some 17.0.2 tests. Great that files uploaded via browser now consistently keep their last modified info. Why is it different when using file drop though?

[wrightsinreston]

Thanks for documenting this issue. I do not use a browser or WebDAV. Rather, I connect Win-10, Win-7, and Ubuntu clients to a NC v17 server running on a Raspberry Pi using the NC client apps. All the clients use the v2.6.3 NC client app appropriate for their OS architecture. When I change a synchronized file on any client, the NC server correctly synchronizes the updated content across all hosts as expected and the NC Web Admin page and Windows hosts all report the correct last-modified time. But the Ubuntu host reports the time at which the server synchronized it, which will differ for each Ubuntu client. This really confuses some java apps I've written which rely on last-modified time to make decisions. As you have noted in your write-up: "This should just work." And, at least, it should work the same way on all platforms. I do not consider this to be an Enhancement request and believe it should be registered as a bug.

[wrightsinreston]

Here is some further information for those encountering this issue. I believe this offers a viable work-around even if the underlying issue appears to remain.

The issue appears to manifest itself only when the following three conditions are met: (1) The Nextcloud client is running on a Linux box, (2) the files in question are NOT owned by the UID of the running Nextcloud client process. (Note: Even if file permissions allow read/write, the file MUST be owned by the UID of the Nextcloud client process,) and (3) the disk volume on which the files in question reside is mounted under a UID different from that of the file owner. If all three of those criteria are met, Nextcloud will be unable to set the Last-Modified time and the time-stamp will reflect the time at which the file was synchronized.

If the disk volume is mounted under the same UID as the running Nextcloud client process (even if it resides on a different host PC) or if it is mounted as a CIFS volume, then the issue vanishes and Nextcloud can successfully set the time-stamp as expected.

This appears to be unique to Linux hosts. Windows hosts do not appear to manifest this issue regardless of file ownership or file system.

[Kuphi]

@despens can you update your table for nextcloud 18 and 19? Thank you! :1st_place_medal:

[despens]

I just updated one of my nextcloud instances to version 19 and updated the above table.

As @MorrisJobke stated in https://github.com/nextcloud/server/issues/20862#issuecomment-625691534 the intended behavior is to display the last-modified time. But that information is easily lost or never makes it into nextcloud, depending on what kind of storage or upload mechanism is used. In many cases, last-modified time will hold uploaded time.

This issue is blocking the use of different storage providers for hot and cold storage: Hot storage are files currently in use and need fast access. This can be local disk or local block storage. Cold storage is for files that are not in active use anymore and can go into an archive, which could be on much cheaper storage offered by a specialized provider. However, when files are moved to cold storage they lose their last-modified dates and appear as recently modified.

I'm looking for a way to help nextcloud at least support moving files from one storage to another while retaining the last-modified date of each file, to enable the hot/cold storage use-case. What's the best way of doing that?

[ltguillaume]

The current Android client (currently running v3.12.1 RC1 F-Droid, with Nextcloud v19.0.0) seems to exhibit totally random behavior with regard to the modification date: using the Auto upload feature, some files are uploading while preserving the modification date, while others aren't. No distinction can be made in this between <10MB and >10MB.

[bodlin]

Hi folks, I just tested copy via cloud federation between two freshly installed Nextcloud servers (both v19.0.0). What I did via web UIs:

1. user1@nc1 shares SrcDir for user2@nc2
2. user2@nc2 accepts invitation for the share
3. user2@nc2 views SrcDir@nc1 with all files and all timestamps correct
4. user2@nc2 performs copy of all files from SrcDir@nc1 to DestDir@nc2
5. user2@nc2 views DestDir@nc2 - timestamps of all files are less than minute ago

For photos is timestamp crucial thing.

[despens]

@bodlin, I added your result to the table above.

[codiflow]

Is there a reason why theres no NC v18 column in the table? I can confirm the issue still exists using WebDAV with NC v18.

[despens]

@codiflow The only reason is that I upgraded from 17 to 19 directly on my instances :wink:. I'll add you info.

[da-anda]

I used the official Android app to sync my pictures to my freshly installed NC20 instance, and can confirm that files > 1M loose their original creation date and instead get the date of the upload/sync. This is super annoying and IMO a no-go. Timestamps should be preserved no matter what. NC is not usable for me this way.

I also do not understand why NC server is unable to extract the file creation time and add this info into the DB on it's own, but this is a different story.

edit: to me this is a clear bug and not an enhancement like this ticket is currently labeled

[satyamkapoor]

Unfortunately Nextcloud Photos is useless because of this bug

[szaimen]

Pretty amazing work! Thank you for collecting and displaying it here :)

[maxxer]

I can confirm rclone -> NC 22 (external storage) does not preserve creation/modification times.

Further info here

[satyamkapoor]

I can confirm rclone -> NC 22 (external storage) does not preserve creation/modification times

It's disheartening. Overly expensive to scale. :(

[ltguillaume]

Does ownCloud handle this properly? Could anything be ported to NC if so? To be honest, it seems to me that Nextcloud keeps expanding into other flashy fields of productivity, while it still fails to reliably offer its core tasks, such as this. It worries me.

[codiflow]

For Nextcloud 22 I can confirm the following situation (I have tested everything with <10 and >10MB files):

Method	Source	Target	Result
file upload > 10 MB via browser	Desktop	Primary Storage	:heavy_check_mark:
file upload < 10 MB via browser	Desktop	Primary Storage	:heavy_check_mark:
upload via file drop	Desktop	Primary storage	:heavy_check_mark:
sync files with Desktop client	Desktop	Primary Storage	:heavy_check_mark:
upload files via WebDAV (@codiflow)	Desktop	Primary Storage	:no_entry:

[maxxer]

For Nextcloud 22 I can confirm the following situation (I have tested everything with <10 and >10MB files): upload files via WebDAV (@codiflow) Desktop Primary Storage ⛔

How strange, I did try uploading a file with rclone (webdav) to a local storage directory and mod time was preserved.

[ltguillaume]

@maxxer RClone supports ownCloud/Nextcloud explicitly for this using an *MTIME header, I guess you chose "Nextcloud" when setting up the storage configuration?

[maxxer]

@maxxer RClone supports ownCloud/Nextcloud explicitly for this using an *MTIME header, I guess you chose "Nextcloud" when setting up the storage configuration?

Yes, sure!

[bjornidar]

For Nextcloud 23 I can confirm the following: Method Source Target Result sync files with Desktop client Desktop Primary Storage ⛔ sync files with Browser Desktop Primary Storage ✔️