rclone icon indicating copy to clipboard operation
rclone copied to clipboard
verified publisher icon rclone

--skip-links no longer works on Windows

Open traynier opened this issue 9 months ago • 4 comments

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/skip-links-no-longer-works-on-windows/51083/1

What is the problem you are having with rclone?

--skip-links no longer works on Windows.

This was broken in version 1.69.2. It works in 1.69.1 and earlier.

What is your rclone version (output from rclone version)

rclone v1.69.2

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Microsoft Windows 11 Pro 24H2 24H2 (64 bit)

Which cloud storage system are you using? (e.g. Google Drive)

Local storage

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone copy C:\Users\Public\ C:\00\ --skip-links --retries=1

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

Without -vv it fails on the links:

2025/05/21 20:57:39 ERROR : Documents/My Music: Failed to copy: failed to open source object: Access is denied.
2025/05/21 20:57:39 ERROR : Documents/My Pictures: Failed to copy: failed to open source object: Access is denied.
2025/05/21 20:57:39 ERROR : Documents/My Videos: Failed to copy: failed to open source object: Access is denied.
2025/05/21 20:57:39 ERROR : Attempt 1/1 failed with 3 errors and: failed to open source object: Access is denied.
2025/05/21 20:57:39 NOTICE: Failed to copy with 3 errors: last error was: failed to open source object: Access is denied.

With -vv doesn't anything interesting:

2025/05/21 20:56:50 DEBUG : rclone: Version "v1.69.2" starting with parameters ["C:\\Users\\Paul\\AppData\\Local\\Microsoft\\WinGet\\Packages\\Rclone.Rclone_Microsoft.Winget.Source_8wekyb3d8bbwe\\rclone-v1.69.2-windows-amd64\\rclone.exe" "-vv" "copy" "C:\\Users\\Public\\" "C:\\00\\" "--skip-links" "--retries=1"]
2025/05/21 20:56:50 DEBUG : Creating backend with remote "C:\\Users\\Public\\"
2025/05/21 20:56:50 DEBUG : Using config file from "C:\\Users\\Paul\\.config\\rclone\\rclone.conf"
2025/05/21 20:56:50 DEBUG : local: detected overridden config - adding "{HK82T}" suffix to name
2025/05/21 20:56:50 DEBUG : fs cache: renaming cache item "C:\\Users\\Public\\" to be canonical "local{HK82T}://?/C:/Users/Public"
2025/05/21 20:56:50 DEBUG : Creating backend with remote "C:\\00\\"
2025/05/21 20:56:50 DEBUG : local: detected overridden config - adding "{HK82T}" suffix to name
2025/05/21 20:56:50 DEBUG : fs cache: renaming cache item "C:\\00\\" to be canonical "local{HK82T}://?/C:/00"
2025/05/21 20:56:50 DEBUG : AccountPictures: Directory modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Added delayed dir = "AccountPictures", newDst=AccountPictures
2025/05/21 20:56:50 DEBUG : Desktop: Directory modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Added delayed dir = "Desktop", newDst=Desktop
2025/05/21 20:56:50 DEBUG : Documents: Directory modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Added delayed dir = "Documents", newDst=Documents
2025/05/21 20:56:50 DEBUG : Downloads: Directory modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Added delayed dir = "Downloads", newDst=Downloads
2025/05/21 20:56:50 DEBUG : Libraries: Directory modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Added delayed dir = "Libraries", newDst=Libraries
2025/05/21 20:56:50 DEBUG : Music: Directory modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Added delayed dir = "Music", newDst=Music
2025/05/21 20:56:50 DEBUG : Pictures: Directory modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Added delayed dir = "Pictures", newDst=Pictures
2025/05/21 20:56:50 DEBUG : Videos: Directory modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Added delayed dir = "Videos", newDst=Videos
2025/05/21 20:56:50 DEBUG : desktop.ini: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : desktop.ini: Unchanged skipping
2025/05/21 20:56:50 DEBUG : Downloads/desktop.ini: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001: Directory modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/desktop.ini: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Desktop/desktop.ini: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Desktop/desktop.ini: Unchanged skipping
2025/05/21 20:56:50 DEBUG : Documents/My Music: Need to transfer - File not found at Destination
2025/05/21 20:56:50 DEBUG : Music/desktop.ini: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Music/desktop.ini: Unchanged skipping
2025/05/21 20:56:50 DEBUG : Libraries/desktop.ini: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Libraries/desktop.ini: Unchanged skipping
2025/05/21 20:56:50 DEBUG : Libraries/RecordedTV.library-ms: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Libraries/RecordedTV.library-ms: Unchanged skipping
2025/05/21 20:56:50 DEBUG : Pictures/desktop.ini: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Pictures/desktop.ini: Unchanged skipping
2025/05/21 20:56:50 DEBUG : Videos/desktop.ini: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Videos/desktop.ini: Unchanged skipping
2025/05/21 20:56:50 DEBUG : Downloads/desktop.ini: Unchanged skipping
2025/05/21 20:56:50 DEBUG : Added delayed dir = "AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001", newDst=AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001
2025/05/21 20:56:50 ERROR : Documents/My Music: Failed to copy: failed to open source object: Access is denied.
2025/05/21 20:56:50 DEBUG : AccountPictures/desktop.ini: Unchanged skipping
2025/05/21 20:56:50 DEBUG : Documents/My Pictures: Need to transfer - File not found at Destination
2025/05/21 20:56:50 DEBUG : Documents/My Videos: Need to transfer - File not found at Destination
2025/05/21 20:56:50 DEBUG : Documents/desktop.ini: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Documents/desktop.ini: Unchanged skipping
2025/05/21 20:56:50 ERROR : Documents/My Pictures: Failed to copy: failed to open source object: Access is denied.
2025/05/21 20:56:50 ERROR : Documents/My Videos: Failed to copy: failed to open source object: Access is denied.
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image1080.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : Local file system at //?/C:/00: Waiting for checks to finish
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image1080.jpg: Unchanged skipping
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image48.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image48.jpg: Unchanged skipping
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image64.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image64.jpg: Unchanged skipping
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image208.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image240.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image32.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image40.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image424.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image448.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image192.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image96.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image208.jpg: Unchanged skipping
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image240.jpg: Unchanged skipping
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image32.jpg: Unchanged skipping
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image40.jpg: Unchanged skipping
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image424.jpg: Unchanged skipping
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image448.jpg: Unchanged skipping
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image192.jpg: Unchanged skipping
2025/05/21 20:56:50 DEBUG : AccountPictures/S-1-5-21-3762022837-651421538-212557348-1001/{F0A8606B-337F-47BD-B74E-15A6499AEFF1}-Image96.jpg: Unchanged skipping
2025/05/21 20:56:50 DEBUG : Local file system at //?/C:/00: Waiting for transfers to finish
2025/05/21 20:56:50 INFO  : Documents: Set directory modification time (using SetModTime)
2025/05/21 20:56:50 ERROR : Attempt 1/1 failed with 3 errors and: failed to open source object: Access is denied.
2025/05/21 20:56:50 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 3 (retrying may help)
Checks:                21 / 21, 100%
Elapsed time:         0.0s

2025/05/21 20:56:50 DEBUG : 4 go routines active
2025/05/21 20:56:50 NOTICE: Failed to copy with 3 errors: last error was: failed to open source object: Access is denied.

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

traynier avatar May 21 '25 20:05 traynier

Bisect indicates this is the commit which broke it 775f3e49f0b03f7931a2115fab86706134811b72 which seems unlikely, but reverting it does fix the problem.

So upgrading one of these packages caused the problem

-       golang.org/x/crypto v0.31.0
+       golang.org/x/crypto v0.35.0
-       golang.org/x/sync v0.10.0
+       golang.org/x/sync v0.11.0
-       golang.org/x/sys v0.28.0
+       golang.org/x/sys v0.30.0
-       golang.org/x/text v0.21.0
+       golang.org/x/text v0.22.0
-       golang.org/x/term v0.27.0
+       golang.org/x/term v0.29.0

After a bit of experimentation it looks like this is the culprit

-       golang.org/x/sys v0.30.0
+       golang.org/x/sys v0.31.0

Upgrading x/sys to the latest v0.33.0 does not fix the problem alas so we'll have to dig in deeper.

This bit of code is makes the decision as to whether to skip symlinks or not.

https://github.com/rclone/rclone/blob/b8fde4fc46f04fe2512e95565d22331ff36b16a7/backend/local/local.go#L1200-L1204

So the mode must have changed between the versions.

Good

Z:\go\src\github.com\rclone\rclone>rclone lsf --skip-links c:\users\dev 2>&1 | find "My"
2025/05/22 11:05:12 NOTICE: My Documents: mode = Lrw-rw-rw- (0x080001b6) os.ModeSymlink=0x08000000 mode&os.ModeSymlink != 0 = true

Bad

Z:\go\src\github.com\rclone\rclone>rclone lsf --skip-links c:\users\dev 2>&1 | find "My"
2025/05/22 11:04:11 NOTICE: My Documents: mode = ?rw-rw-rw- (0x000801b6) os.ModeSymlink=0x08000000 mode&os.ModeSymlink != 0 = false
My Documents

So the mode has lost its symlink status between golang.org/x/sys v0.30.0 and v0.31.0

It turns out after a lot of investigation that the problem is caused by updating the minimum go version in the go.mod from go1.21 to go1.22

With go 1.21 in go.mod success of a simple program which does os.Lstat and prints the results

Z:\go\src\github.com\rclone\rclone>z:\Go\stattest\stattest.exe "c:\users\dev\My Documents"
Name = "My Documents"
Size = 0
Mode = 0x80001B6 (Lrw-rw-rw-)
ModTime = 2021-07-16 00:44:42.7917375 +0100 BST
IsDir = false
IsSymlink = true

With go 1.22 in go.mod - failure. The symlink bit appears to be in the wrong place.

Z:\go\src\github.com\rclone\rclone>z:\Go\stattest\stattest.exe "c:\users\dev\My Documents"
Name = "My Documents"
Size = 0
Mode = 0x801B6 (?rw-rw-rw-)
ModTime = 2021-07-16 00:44:42.7917375 +0100 BST
IsDir = false
IsSymlink = false

I've made an upstream bug in Go about this https://github.com/golang/go/issues/73827 as this is quite unexpected.

PS the upgrade to the go statement in the go.mod was forced by the security updates to x/crypto.

ncw avatar May 22 '25 11:05 ncw

It turns out that this behaviour is as expected and we will have to find a workaround.

I've had a go at one here - can you give it a try?

v1.70.0-beta.8742.b196f1105.fix-8561-windows-symlinks on branch fix-8561-windows-symlinks (uploaded in 15-30 mins)

ncw avatar May 22 '25 16:05 ncw

I've tried the fix and it solved the issues for me.

Thank you very much!

jcasale avatar May 22 '25 22:05 jcasale

Wow, that was quite the deep dive, and rather unexpected change in golang!

But yes, I can confirm --skip-links is working in that beta. Many thanks!!

traynier avatar May 23 '25 16:05 traynier

This should be all working in v1.70 so I will close this.

ncw avatar Jun 19 '25 17:06 ncw

I didn't get around to testing 1.70.0, however I have tried 1.70.0 and 1.70.1, and there is a regression compared to the fix posted above. Both release builds behave like 1.69.2.

jcasale avatar Jun 20 '25 19:06 jcasale

@ncw Yes, unfortunately I'm seeing the same as @jcasale - although it was fixed in the 1.70 beta, it is still broken in 1.70 and 1.70.1

traynier avatar Jun 22 '25 09:06 traynier

It looks like branch https://github.com/rclone/rclone/tree/fix-8561-windows-symlinks was not merged back to master?

traynier avatar Jun 22 '25 09:06 traynier

@traynier @jcasale yes I forgot to merge the branch - sorry :-(

I've merged into the latest beta now and I'll release it in v1.71 and v1.70.2

ncw avatar Jun 23 '25 15:06 ncw

This has now been released in v1.70.2

ncw avatar Jun 27 '25 13:06 ncw

Hi @ncw!

First of all, thank you very much for the great work! I am coming from the forum post https://forum.rclone.org/t/windows-build-ignoring-copy-links-in-rclone-1-70/52187

This seems like a very similar bug to the one that is happening to me (with --copy-links flag).

forcegk avatar Aug 07 '25 08:08 forcegk

@forcegk Can you please open a new issue? You can link to this issue from there. Please avoid bumping closed issues because they are not monitored.

darthShadow avatar Aug 07 '25 16:08 darthShadow