Files with same size but different timestamp/checksum are not synced

[kaoneko]

Edit: Read this comment for a TLDR; and analysis as of May 5th.

---

• [✔️] I have read the above note

What version of Round Sync are you using (About -> App version)?

2.2.3-terrapin

What is your Android version, phone model and manufacturer?

Android 13, Google Pixel 4a

Which steps are required to reproduce this issue?

I noticed some daily backups were not updated on my remote. They are files with the same size but different contents and a different timestamp. Activating Use md5 checksum made no difference. I'll walk you through my setup, but I don't know if it's specific to a Crypt remote on a WebDAV remote. #37 implies this topic still needs investigation.

1. Create a WebDAV remote
2. Create a Crypt remote pointing to a folder on the WebDAV remote
3. Create a local folder with a text file with contents abc
4. Create the same folder and file on the Crypt remote but make the contents of the file 123
5. Check that both files have the same size but different timestamps
6. Create a task that syncs the local folder to the remote folder ("Sync local to remote", with or without "Use md5 checksum")
7. Run the task, the result is "There was nothing to do" and the local file is not copied to the remote

Please also enable rclone logging (Settings > Logging > Log Rclone errors). You're going to need the log for the last question.

No log file was written after enabling this setting and running the task. After force closing and restarting Round Sync and running the task, a log file was written though. It's one of the bugs I alluded to in my recent report.

Despite your statement, the log file was not needed to answer the last question.

What is your configuration (rclone.conf)?

Note: these are not my real credentials and webdav host.

[Redacted]
type = webdav
rclone_remote_name = Redacted
pass = hunter2
user = AzureDiamond
url = https://webdav.redacted.com

[backup]
type = crypt
password = hunter2hunter2
directory_name_encryption = true
remote = Redacted:backup

[3d11ef8a-387b-3205-e52f-26e3c7e172e6]
type = alias
remote = /storage/emulated/0

Does the same issue also occur when using the same configuration on a PC or in Termux?

Tested on PC with rclone sync /local/folder/ backup:folder/: local file was copied to remote Tested on Android with S3Drive: local file was copied to remote

[kaoneko]

Alright, I've looked into this some more and found out a few things:

• Taking the crypt out of the equation doesn't influence the bug
• The issue doesn't occur with a Koofr remote
• On the Koofr remote I can tap to calculate the MD5 or SHA1 checksum of a file in its Properties popup, the first returns a checksum, the second UNSUPPORTED; on the WebDAV remote both return ERROR

Now the rclone documentation says:

Plain WebDAV does not support modified times. However when used with Fastmail Files, Owncloud or Nextcloud rclone will support modified times.

Likewise plain WebDAV does not support hashes, however when used with Fastmail Files, Owncloud or Nextcloud rclone will support SHA1 and MD5 hashes.

So now it makes sense right? I guess this bug is not solvable for plain WebDAV remotes then?

However, I'm using Fastmail. But as you can see in the exported configuration in my opening post, the [Redacted] entry is missing the line vendor = fastmail. This remote was configured using Round Sync, including selecting the vendor type (which, like some other fields, also doesn't show a value when you edit the remote again, by the way).

The rclone.conf on my PC does have the vendor line. So I added it to the config file in the Round Sync export and imported it. And what do you know? Syncing a file with the same size but different timestamp/checksum on local vs remote now works when Use md5 checksum is enabled in the task! Yay.

At the same time, the Tap to calculate MD5 or SHA1 still both return an error for most files I tested it with. I've got one file which does show a SHA1 checksum somehow...

I also encountered yet another bug when figuring out this one: when importing a Round Sync backup which has md5sum: true in the JSON of every task in rcx.json, all imported tasks will actually have Use md5 checksum disabled, and when you then export a new config backup, this is also reflected in the JSON.

At this point I am somewhat shocked at the sheer amount of severe bugs (bugs that may lead to data loss) I've already encountered in Round Sync, i.e. this one, using the checksum being disabled on import, scheduled Triggers not running, a new remote overwriting an existing remote without warning when it has the same name. Add to that the many other weird bugs and I'm not sure if I can trust this software with my data.

Is this something you guys are aware of? I see a new "stable" version has just been released, I haven't checked it out yet. Is it a step forward?

The beta releases have a clear warning on them. I think the 2.2.3 version should've also had that warning and the beta label, because it's not stable at all in my book.

I really think this is app has potential, and there's already so many things it does right as far as I can see, like the clean interface, being able to configure remotes in the app, scheduling (if it can be trusted) and the content provider! So I'm not here to criticize it, but to help it get better, because I really want it to be better!

Love to hear your thoughts, devs & contributors ❤️

[newhinton]

Thanks for looking into those issues, that is very helpful to me.

Let me first adress the comment regarding existing bugs and beeing aware of them:

I am more or less the sole developer of this app, and i am certainly not aware of many bugs. My time is very limited, and there is a lot of old code and required maintenance, so writing tests hasn't come up. I am relying on people like you to report specific issues, that i then can adress.

Now to the rest:

If i understand, this is three seperate issues:

1. Remote creation overwriting existing remotes
2. Remote editing not listing formerly changed values (could you list the fields you did find that were not properly repopulating?)
3. Import/Export not properly respecting md5sum

The other issues are already seperate tickets. Does this properly summarize your findings?

[mtalexan]

It sounded to me like the root of the specific problem was the vendor not being set in the actual rclone config file for the WebDav type when it was set in the Round Sync GUI.

Also it sounds like maybe the Calculate checksum option on files viewed from the built in remote browser(?) errors out rather than calculating the checksum if the endpoint type doesn't support timestamps (?).

[kaoneko]

@newhinton

I am more or less the sole developer of this app, and i am certainly not aware of many bugs. My time is very limited, and there is a lot of old code and required maintenance, so writing tests hasn't come up. I am relying on people like you to report specific issues, that i then can adress.

Fair enough, I'll see if I can add some solid bug reports on the issues I encountered. Thank you for your work on Round Sync 🙏🏻

If i understand, this is three seperate issues:

1. Remote creation overwriting existing remotes

Correct. I'll test if this still happens in the latest version and open an issue about this.

2. Remote editing not listing formerly changed values (could you list the fields you did find that were not properly repopulating?)

This doesn't seem to be an issue anymore in the current release 👍🏻

3. Import/Export not properly respecting md5sum

Correct. I'll do a proper bug report in a separate issue.

The other issues are already seperate tickets. Does this properly summarize your findings?

Yes.

@mtalexan

It sounded to me like the root of the specific problem was the vendor not being set in the actual rclone config file for the WebDav type when it was set in the Round Sync GUI.

Yes, this must be causing the issue for WebDAV remotes with a vendor that supports hashes and modification times.

Also it sounds like maybe the Calculate checksum option on files viewed from the built in remote browser(?) errors out rather than calculating the checksum if the endpoint type doesn't support timestamps (?).

Yes, Calculate checksum was used from the built-in remote browser. It errors out even after importing an rclone.conf with vendor = fastmail, apart from the SHA1 checksum on one particular (and particularly large [440 MB]) file...

Not solved for crypt remote

In the meantime I found out that, after adding the vendor = fastmail line to Round Sync, the syncing a newer file with the same size issue is NOT solved when syncing to the crypt hosted on my WebDAV remote. As stated before, this issue does not occur with rclone on my laptop (which also uses rclone 1.66) nor with another rclone client for Android.

Possibly relevant pieces of documentation on https://rclone.org/crypt/:

Modification times and hashes

Crypt stores modification times using the underlying remote so support depends on that.

Hashes are not stored for crypt. However the data integrity is protected by an extremely strong crypto authenticator.

Use the rclone cryptcheck command to check the integrity of an encrypted remote instead of rclone check which can't check the checksums properly.

and

Rclone does not encrypt

• file length - this can be calculated within 16 bytes
• modification time - used for syncing

[kaoneko]

I'm not good with walls of text, so here's a recap for clarity.

Issue: Local file with same size as remote file but newer modification time and different hash/content is not uploaded to remote when syncing local to remote.

	Use md5 checksum	WebDAV	crypt-on-WebDAV
rclone.conf generated through app	off	issue	issue
	on	issue	issue
rclone.conf with vendor = fastmail	off	issue	issue
	on	solved	issue

Todo:

• [ ] fix vendor type not added to rclone.conf
• [ ] find out why issue still occurs on crypt

Related issue: Calculating checksums in file browser displays ERROR on WebDAV remote (for most files); not fixed by vendor = fastmail.

[ndesgranges]

Likewise plain WebDAV does not support hashes, however when used with Fastmail Files, Owncloud or Nextcloud rclone will support SHA1 and MD5 hashes.

I'm using plain webdav and this statement is not to my liking 😄

Why is rclone / round-Sync relying on server side checksum ? It might be very slightly slower to do everything locally, but this way most of the issues above should be fixed right ? (just wondering)