backend: Add Apple iCloud Drive backend

[lostb1t]

What is the purpose of this change?

Add support for a new backend, iCloud Drive

Was the change discussed in an issue or in the forum before?

#1778

Checklist

• [x] I have read the contribution guidelines.
• [x] I have added tests for all changes in this PR if appropriate.
• [x] I have added documentation for the changes if appropriate.
• [x] All commit messages are in house style.
• [x] I'm done, this Pull Request is ready for review :-)

Notes

This feature branch is in a working state, feedback on finished implementations is appreciated. Ill keep it in draft untill everything is implemented and ill update the TODO accordingly

This PR is only for iCloud Drive. Ill continue work on Photos once this PR is merged.

Todo

Rclone Interfaces

Fs

• [x] List
• [x] Mkdir
• [x] Rmdir
• [x] Put

Object

• [x] Open
• [x] Update
• [x] Remove

Mover

• [x] Move

DirMove

• [x] DirMove

Copier

• [x] Copy

[ncw]

Nice work :-)

Some quick questions

• where did the client_id come from?
• are you running through the integration tests?

Thank you

[lostb1t]

Nice work :-)

Some quick questions

• where did the client_id come from?
• are you running through the integration tests?

Thank you

not sure what client_id you are referring to? and no I havent run any integration test yet. Will do whenimplemented all interfaces.

[lostb1t]

fyi: so iCloud doesn't have functionality for updating/replacing files. At least not something I can find reverse engineering

So if a file exists I move it to iCloud trash. And upload a new version of the file. After successful upload the file is removed from trash or restored from trash if upload failed.

[lostb1t]

@ncw how one handle folder invalidation? drive uses etags for files and folders. If a file or folder within a folder changes then the parent folders etag changes. This means I have to clear almost the whole tree from child to parent.

Not sure how to handle that in general, what if a users change it locally but file/folder has changed remotely in between. What error should I raise on that? (if etags are not matching)

cachedir doesn't seem to refresh after invalidation, see: https://github.com/rclone/rclone/blob/37816e9fbafdb8eee1483a7eab34aa2ff9b1f191/backend/icloud/icloud.go#L424

When calling flushdir at the end of rmdir I see the call for FindLeaf but the cachedir entries aren't updated.

[ncw]

@lostb1t

not sure what client_id you are referring to?

This one at the top of the source!

clientId      = "e9f98057fb916de2bbd755ef280d7257146a76e5118f27ab2e9a3d065c20c17e"

and no I haven't run any integration test yet. Will do when implemented all interfaces.

Running the integration tests will guide you through what you need to do. You can start running them as soon as you've done List.

fyi: so iCloud doesn't have functionality for updating/replacing files. At least not something I can find reverse engineering

So if a file exists I move it to iCloud trash. And upload a new version of the file. After successful upload the file is removed from trash or restored from trash if upload failed.

Seems OK!

how one handle folder invalidation? drive uses etags for files and folders. If a file or folder within a folder changes then the parent folders etag changes. This means I have to clear almost the whole tree from child to parent. Not sure how to handle that in general, what if a users change it locally but file/folder has changed remotely in between. What error should I raise on that? (if etags are not matching)

Do you have to use the etags? They are optional for most cloud storage systems and rclone doesn't use them. This means that the last write wins which is what we want. If you can't ignore etags then refresh the etag and try again.

cachedir doesn't seem to refresh after invalidation, see:

https://github.com/rclone/rclone/blob/37816e9fbafdb8eee1483a7eab34aa2ff9b1f191/backend/icloud/icloud.go#L424

When calling flushdir at the end of rmdir I see the call for FindLeaf but the cachedir entries aren't updated.

Not sure!

[lostb1t]

@lostb1t

not sure what client_id you are referring to?

This one at the top of the source!

clientId      = "e9f98057fb916de2bbd755ef280d7257146a76e5118f27ab2e9a3d065c20c17e"

and no I haven't run any integration test yet. Will do when implemented all interfaces.

Running the integration tests will guide you through what you need to do. You can start running them as soon as you've done List.

fyi: so iCloud doesn't have functionality for updating/replacing files. At least not something I can find reverse engineering So if a file exists I move it to iCloud trash. And upload a new version of the file. After successful upload the file is removed from trash or restored from trash if upload failed.

Seems OK!

how one handle folder invalidation? drive uses etags for files and folders. If a file or folder within a folder changes then the parent folders etag changes. This means I have to clear almost the whole tree from child to parent. Not sure how to handle that in general, what if a users change it locally but file/folder has changed remotely in between. What error should I raise on that? (if etags are not matching)

Do you have to use the etags? They are optional for most cloud storage systems and rclone doesn't use them. This means that the last write wins which is what we want. If you can't ignore etags then refresh the etag and try again.

cachedir doesn't seem to refresh after invalidation, see: https://github.com/rclone/rclone/blob/37816e9fbafdb8eee1483a7eab34aa2ff9b1f191/backend/icloud/icloud.go#L424

When calling flushdir at the end of rmdir I see the call for FindLeaf but the cachedir entries aren't updated.

Not sure!

etags are needed for file operations unfortunately, but I can just refresh it when it got stale.

the clientid is a static that icloud.com uses. So I reuse it to mimic the browser.

[lostb1t]

@ncw small question. iCloud Drive does not allow uploading empty files so check the size and raise ErrorCantUploadEmptyFiles when its 0

But when using macOS finder to copy a file, I get an partial file with zero length (while the actually file isn't empty)

Not sure howto handle this.

[ncw]

@lostb1t

small question. iCloud Drive does not allow uploading empty files so check the size and raise ErrorCantUploadEmptyFiles when its 0

That sounds correct.

But when using macOS finder to copy a file, I get an partial file with zero length (while the actually file isn't empty)

Are you using rclone mount in the finder?

So you are seeing a zero length file in the finder? I'm not sure what you mean by a partial file?

Can you reproduce this without the rclone mount? So using rclone copy? If so please paste your command line and the log with -vv.

[lostb1t]

Its slow but finished alle interfaces and all integration test are passing :tada:

Only documentation left before its ready for review.

[ncw]

Its slow but finished all interfaces and all integration test are passing 🎉

Well done!

You are running the full suite from test_all?

Only documentation left before its ready for review.

:-)

[epetousis]

@lostb1t not sure if it's too early for bug reports, but currently it looks like app folders aren't being represented as folders, but rather as files.

[lostb1t]

@lostb1t not sure if it's too early for bug reports, but currently it looks like app folders aren't being represented as folders, but rather as files.

bug reports are welcome. Will fix with the following pass, tnx

[lostb1t]

@epetousis this has been fixed

[lostb1t]

crap i broke my icloud. Too many trash items and now the endpoint is broken lol

the icloud web api is a mess

[epetousis]

Fix works great. Much appreciated!

[cyayon]

Hi, I setup successfully and authentication seems ok, but I do not see any files or directories on my iCloud account. I tried ls, lsl, lsd... no file.

thanks.

[lostb1t]

Hi,

I setup successfully and authentication seems ok, but I do not see any files or directories on my iCloud account.

I tried ls, lsl, lsd... no file.

thanks.

@cyayon Could you post a log from the command with the -vv flag (e.g. output from rclone -vv ls

[cyayon]

Hi,

here is the verbose.

./rclone-icloud version
rclone v1.67.0-DEV
- os/version: arch (64 bit)
- os/kernel: 6.8.5-arch1-1 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.2
- go/linking: dynamic
- go/tags: none

./rclone-icloud -vvv --config rclone-icloud.conf lsd icloud:/
2024/04/20 11:04:04 DEBUG : rclone: Version "v1.67.0-DEV" starting with parameters ["./rclone-icloud" "-vvv" "--config" "rclone-icloud.conf" "lsd" "icloud:/"]
2024/04/20 11:04:04 DEBUG : Creating backend with remote "icloud:/"
2024/04/20 11:04:04 DEBUG : Using config file from "/home/pkg/pkgbuild/rclone/rclone-icloud/rclone/rclone-icloud.conf"
2024/04/20 11:04:04 DEBUG : icloud: Valid session, no need to reauth
2024/04/20 11:04:04 DEBUG : fs cache: renaming cache item "icloud:/" to be canonical "icloud:"
2024/04/20 11:04:05 DEBUG : 8 go routines active

./rclone-icloud -vvv --config rclone-icloud.conf ls icloud:/
2024/04/20 11:04:58 DEBUG : rclone: Version "v1.67.0-DEV" starting with parameters ["./rclone-icloud" "-vvv" "--config" "rclone-icloud.conf" "ls" "icloud:/"]
2024/04/20 11:04:58 DEBUG : Creating backend with remote "icloud:/"
2024/04/20 11:04:58 DEBUG : Using config file from "/home/pkg/pkgbuild/rclone/rclone-icloud/rclone/rclone-icloud.conf"
2024/04/20 11:04:59 DEBUG : icloud: Valid session, no need to reauth
2024/04/20 11:04:59 DEBUG : fs cache: renaming cache item "icloud:/" to be canonical "icloud:"
2024/04/20 11:04:59 DEBUG : 8 go routines active

[lostb1t]

do you maybe have advanced data protection enabled?

[cyayon]

Yes I have. Is there an issue with that ?

[lostb1t]

yes its currently unsupported

[cyayon]

Ok, I understand. Do you plan to support it or is it too complicated to implement ? Thanks

[lostb1t]

i hope to add support for it eventually.Its a different flow, need to dive in a bit to see what it entails

[cyayon]

Thanks !--Christophe YayonOn 20 Apr 2024, at 12:56, lostb1t @.***> wrote: i hope to add support for it eventually.Its a different flow, need to dive in a bit to see what it entails

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[cyayon]

I just disabled the advanced protection and it do not change everything. No file or directory found ... Same verbose infos.

[lostb1t]

I just disabled the advanced protection and it do not change everything. No file or directory found ... Same verbose infos.

try to delete the remote and add it again (reauthenticate)

[cyayon]

Great ! it works. Thanks

[cescofry]

Works for me on ArchLinux 6.8.7-1.

I had an issue with 2fa, when prompted none of my devices would show the code on screen. However I managed to get it working by going on my iPhone to settings->AppleID->Sigh-in & Security->Two-Factor Authentication->Get Verification Code.

I also struggled when looking for iCloud in the list of rclone remotes as it has not be inserted in the right alphabetical spot.

Great work @lostb1t, cannot wait to see this landed on main. If you need some help to finish this work or just for some testing let me know.

[cyayon]

Hi,

Everything seems to work as expected for a few days. But today, while trying to synchronise just uploaded files to my iCloud Drive, I got an error on rclone sync command :

20/4/04/25 15:58:27 DEBUG : pacer: low level retry 3/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:27 DEBUG : pacer: low level retry 3/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:29 DEBUG : pacer: low level retry 3/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:31 DEBUG : pacer: low level retry 3/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:33 DEBUG : pacer: low level retry 4/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:35 DEBUG : pacer: low level retry 4/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:37 DEBUG : pacer: low level retry 4/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:39 DEBUG : pacer: low level retry 4/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")

Tried to re-launch multiple times, it is not better. any idea ?

could it be the same error ? https://www.macworld.com/article/609008/apps-icloud-fix-service-unavailable-icloud.html

thanks.

[lostb1t]

@cyayon does listing etc your drive works? or do all file operations return a 503?