Advanced Data Protection Support

[gsong]

trafficstars

Describe the bug

$ docker compose exec photos-sync icloud-photos-sync token
-----------------------------------------------------------------------------------------------
Welcome to icloud-photos-sync, v.1.0.1!
Made with <3 by steilerDev
-----------------------------------------------------------------------------------------------
Authenticating user...
Device trusted
Sign in successful!
-----------------------------------------------------------------------------------------------
Experienced fatal error at 3/3/2023, 12:58:54 PM: TokenError (FATAL): Unable to get trust token caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unable to get iCloud Photos service ready caused by iCloudError (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (Error Code: 734410aa-a8db-48ce-971f-0f6041f8751a)
-----------------------------------------------------------------------------------------------

Note that I do have Advanced Data Protection turned on.

Logs Please paste the log file (preferably with LOG_LEVEL=debug), located in .icloud-photos-sync.log, stored in the DATA_DIR.

[2023-03-03T20:58:50.110Z] INFO i-Cloud: Initiating iCloud connection
[2023-03-03T20:58:50.652Z] INFO i-Cloud: Authenticating user
[2023-03-03T20:58:51.323Z] INFO i-Cloud: Authentication successful
[2023-03-03T20:58:51.328Z] INFO i-Cloud: Setting up iCloud connection
[2023-03-03T20:58:53.679Z] INFO i-Cloud: Getting iCloud Photos Service ready
[2023-03-03T20:58:54.627Z] ERROR Error-Handler: TokenError (FATAL): Unable to get trust token caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unable to get iCloud Photos service ready caused by iCloudError (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (Error Code: 734410aa-a8db-48ce-971f-0f6041f8751a)

Operating environmnent

• OS: macOS
• Version: 13.2.1
• Execution environment: docker

[steilerDev]

@gsong are you using iCloud Shared Photo Library?

[gsong]

@gsong are you using iCloud Shared Photo Library?

No, I'm not.

[krubenok]

I'm in a similar position with advanced data protection and a yubikey on my Apple ID. With this combination of security settings, the sign in prompts I receive on other devices don't have MFA codes, but rather just "ok" or "that wasn't me".

Edit; this feels like something that might have a soft dependency on #120 to do yubikey auth via a webUI.

[steilerDev]

@gsong Any of those things enabled with you as well?

~~Otherwise your use case @krubenok would be a separate issue - however: I currently don't have an account setup like this (neither do I plan to do so). Most importantly, in a scenario, where Advanced Data Protection is enabled, the possibility to access your data from the WebUI needs to be enabled (this can be done optionally) - as this tool is using the APIs used by the WebUI.~~

~~In case you want me to investigate your use case further, I'd need an HAR file from your authentication against the iCloud WebUI - based on that I might be able to understand what needs to change in order to support this - #120 is not related. (Full disclosure: Keep in mind that this HAR file might contain sensitive data - unless you know how to purge it, you need to trust me that I won't abuse this - however since the MFA trust token is location/IP specific I probably won't be able to use the data from those requests anyway)~~ See #207

[steilerDev]

I just realise I was skipping over this part @gsong

Note that I do have Advanced Data Protection turned on.

You will need to make sure that access through the iCloud WebUI is enabled. See Apple's support document on this. The tool is re-using those APIs (and I hope the APIs are the same when this is enabled, since I cannot test this).

Please report back - in case it does not work, I would need to ask the same of you as above.

[steilerDev]

@krubenok I've created #207 for the addition of YubiKey support - but I'll need help on that.

[noah-guillory]

Hiya, I'm trying to set this tool up and I have Advanced Data Protection enabled on my iCloud account. I'm currently getting the following error when I try running the token command:

APP_TOKEN (FATAL): Unable to acquire trust token caused by AUTH_FAILED (FATAL): iCloud Authentication failed caused by ICLOUD_PHOTOS_SETUP_FAILED (FATAL): Unable to get iCloud Photos service ready caused by ICLOUD_PHOTOS_SETUP_ERROR (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (Error Code: cadf49b6-85c4-47b4-84a4-ced655765a67)

I've made sure that I have the "Access iCloud Data on the Web` option enabled in my account settings.

Let me know if I could provide you with more information to help troubleshoot this as I do not want to disable ADP.

[steilerDev]

@noah-guillory which 2FA method are you using.

Does the WebUI access work (have you ever accessed the UI from a non Safari browser, where you provided password instead of Touch ID)

[noah-guillory]

@noah-guillory which 2FA method are you using.

Does the WebUI access work (have you ever accessed the UI from a non Safari browser, where you provided password instead of Touch ID)

I am using the normal 2FA method, not using any hardware security keys or anything.

I was able to get through the process of providing my 2FA code by curling it to the MFA endpoint.

And I am able to access Photos from the WebUI using Edge as well. Though whenever I do, I do get a push notification on my Mac saying that my device is providing access to the iCloud web interface.

[steilerDev]

Do you need to confirm this notification before being able to continue?

[steilerDev]

I need to understand how the API behaves differently from the current process, when ADP is enabled.

Best way for me to debug is by being able to see the iCloud API's behaviour here. For that I'd need a HAR file of your login on the browser. For that do the following:

1. Open a new private window in Chrome
2. Navigate to icloud.com
3. Open the Network Tab of the Developer Tools (e.g. right click on the page and select 'Inspect')
4. On the Dev Tools Network select 'Preserve log' and 'Disable cache'
5. Clear the log and make sure logs are recorded
6. Perform login and open iCloud Photos
7. Once done, 'Download har' file
8. Sent it to me so I can take a look at what's happening - feel free to sent it to my email [email protected]

Example of how to do this:

https://user-images.githubusercontent.com/7031616/230869102-3aef6cea-5554-422c-bca2-51bb7f81df6e.mp4

[noah-guillory]

Makes sense! Whenever I get a chance I'll get you that file. Thanks for being responsive 😄

[Tomfox91]

Hi @steilerDev, is there any hope of ADP support landing soon? Did you get the input you needed?

[steilerDev]

@Tomfox91 unfortunately I have not received any feedback on my previous request - so I have not had the chance to implement this.

[steilerDev]

Thanks @Tomfox91 for sending over an HAR file - I just had a quick look - some things look different, but the good news is that the API is very close to what I am expecting :)

Unfortunately I'm not sure when I'll get around working on this as private and professional life are currently taking a lot of time :/ Anyone who wants to support on this, I'm happy to point you into directions :)

[skaeight]

Sadly I think the resolution to this issue is to buy a used / refurb m1 Mac mini.

[steilerDev]

@skaeight looking at the previously shared HAR files by @Tomfox91 I don't think this will be necessary (as long as you allow iCloud Access through the WebUI)

[frprm]

I tried a sync using 1.2.0-beta.4, it fails with this:

Error: APP_SYNC: Sync failed caused by AUTH_FAILED: iCloud Authentication failed caused by ICLOUD_PHOTOS_SETUP_FAILED: Unable to get iCloud Photos service ready caused by ICLOUD_PHOTOS_SETUP_ERROR: Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (error code: 67717c18-aaf8-4d72-9b1c-8b1a66068302)

I can also confirm that "Access iCloud Data on the Web" option is enabled in my account settings.

[steilerDev]

See https://github.com/foxt/icloud.js/issues/4 for some research done on this topic

[github-actions[bot]]

This issue should be resolved with version v1.3.0-beta.1, please confirm.

[steilerDev]

As you see, I've added the initial bits based on the available investigations - I'm pretty sure that it won't work straight away, but I can't test it myself.

So if someone with ADP enabled could run 1.3.0-beta.1 and report back (preferably with a network log), that would be great!

[MaxNeedsSnacks]

Getting a status 500 when the app is trying to request PCS cookies; looking at the network log reveals this response:

                    "content": {
                        "size": 68,
                        "mimeType": "application/json; charset=UTF-8",
                        "text": "{\"success\":false,\"error\":\"Invalid X-APPLE-WEBAUTH-HSA-LOGIN cookie\"}"
                    },

and indeed, looking at the request, I don't think you're setting that cookie correctly...

(above images redacted for obvious reasons)

[steilerDev]

I'm guessing the second screenshot is the reply from setup.icloud.com/setup/ws/1/accountLogin? Meaning we currently don't get a value for the HSA-LOGIN cookie?

[MaxNeedsSnacks]

Sorry, yep, I just noticed that as well, can't set what you don't get :p

Still weird that it gave me no HSA-LOGIN tho, this was on a fresh login if that helps (i.e. no cached trust token or anything)

[steilerDev]

Yeah - on a previously shared HAR file of the web interaction I can see this cookie getting set (and I don't get it for a non-ADP account). So this empty value feels like an application bug - but the HAR file should capture network traffic before processing, unless there is an issue with the underlying axios library parsing the request...

Have you retried to see if this is 'just' a flaky API?

[MaxNeedsSnacks]

Just tried and failed again, specifically tried both logging in again and using an established trust token :/

[steilerDev]

I've checked again - and I do get an WEBAUTH_HSA_LOGIN cookie on the web and none in the tool (but I don't seem to need it as long as I don't have ADP) - so I will need to dig into this...

[steilerDev]

Okay - so it seems the WEBAUTH_HSA_LOGIN token is only temporarily available and necessary to acquire the PCS cookies when ADP is enabled. I think this will require more re-implementation efforts than expected.

Before digging too deep into this: Given the fact, that it seems that ADP enabled accounts need to provide manual authorization to this tool at least every hour (this is how long the PCS cookies seem to be valid for), does it even make sense to support accounts like this with a continuous syncing tool?

I'm curious what the potential users (@MaxNeedsSnacks / @frprm / @noah-guillory / anyone else watching this issue) of this are saying? Especially on the initial run and/or on scheduled runs you'd need to confirm the web login every hour and at every sync - would you (under those constraints) even run this tool (I'm asking this as an honest question, because I wouldn't - and I don't want to waste time on a difficult feature that no one is going to use :D )

[MaxNeedsSnacks]

Hmm.. like I said before in another issue, it works for my usecase personally since the sync would just be occasional and all the authorisation is is a quick button push, but honestly, for continuous sync, I can't imagine many people using this unfortunately ^^;

[steilerDev]

#363 is taking precedence for now - I'll be looking at ADP once I'm back following the current iCloud Web App authentication flow.