[iCloud] Bridge-Thing generates Communication Error

[RalphSester]

The Brigde-Thing (iCloud-Account) does not go online.

It gets the state: COMMUNICATION_ERROR with the following details information:

java.util.concurrent.ExecutionException: org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

The binding is not usable completly at this time.

Current Behavior

The Bridge-Thing does not come online. Restart of binding or OH doesn't help.

Possible Solution

It seems, that apple has changed something in their api.

Steps to Reproduce (for Bugs)

1. Just start the binding.
2. See the error

Environment

• Version used: OpenHAB 3.3 / 3.4
• Operating System: Windows Server 2016

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/icloud-binding-communication-error/122131/336

[BigGeorgeTx]

This is related to this https://github.com/openhab/openhab-addons/issues/13231 previously reported issue, but it appears to have gotten worse. At least for me, A disable/enable cycle, restart of the binding, or restart of openHAB no longer fixes the issue. From the comments in the openHAB community, https://community.openhab.org/t/icloud-binding-communication-error/122131/336?u=biggeorgetx, this is impacting many users.

Tagging those who worked on the last iCloud binding fix just to make sure they are aware of the issue. @lolodomo, @martinvw, @pgfeller, @peteraquino

[JensHoRi]

Maybe this information helps: https://github.com/jetty-project/jetty-reactive-httpclient/issues/33

If I read it right (I' no developer), looks like apple changed something in authentication and is not compliant to specifications anymore. Under that link, they had same issue and discussing a workaround. Looks like doing the authentication handling in code directly could be a temporary good solution. But I'm no developer, maybe I'm wrong. Just try to help

[BigGeorgeTx]

This issue returned today for many users. https://community.openhab.org/t/icloud-binding-communication-error/122131/327?u=biggeorgetx

[RalphSester]

It seems, that the iCloud-Binding has no developer since a couple of months. Nobody seems to care, that the binding is completly unuseable.....

if this problem can't be fixed soon, it would be better to remove it from OH 3.4 !!

@digitaldan: Do you have some ideas?

[martinvw]

I'm not very active currently as a maintainer, and will most likely not be so in the near future because of a planned move, but fixing such issues in an open-source project is not the sole responsibility of the binding maintainers, everyone is allowed, and should make a PR to fix this issue if they are capable to do so.

However, please note that in this specific case we are NOT using an official API provided by the company but relying on some internal API that was never intended for us to use. So it could well be that it is both hard to fix and only a temporary fix.

I do have a short amount of time now to spend on it and I do think that the suggested resolutions are not perse correct because the error tells us that Apple send us a 401 without including a WWW-Authenticate-header, this is not correct according to the spec but IMHO only a symptom of an underlying problem.

We do send authentication in a so-called basic auth header, I just validated in Postman what happens when I perform such a request and it is actively killed by some service in between:

I receive the following response:

HTTP/1.1 401 
Server: AppleHttpServer/3faf4ee9434b
Date: Sun, 09 Oct 2022 09:18:05 GMT
Content-Length: 50
Connection: keep-alive
Set-Cookie: xr_3n2093n1a=HvkfmhuDkyiPeeoYx2YFIDS2DDiRlBJA==; Path=/; Secure; HttpOnly
X-Responding-Instance: fmipservice:3:2299B29512:2844136de5f5
X-Robots-Tag: noindex, noarchive, nosnippet, nofollow
X-Responding-Server: prod-p46-fmipservice--remaining-86888856bf-pm29h_99
X-Responding-Partition: p46
Strict-Transport-Security: max-age=31536000; includeSubDomains;
x-apple-user-partition: 46
via: xrail:mr47p00ic-qujn01123002.me.com:8301:22R479:grp21,631194250daa17e24277dea86cf30319:b2fb3d73b0bb3e9c280a2255c0ac93ae:nlhfd1
X-Apple-Request-UUID: b96e34d2-6603-4c85-b2d8-860866d4733b
access-control-expose-headers: X-Apple-Request-UUID,Via
X-Apple-Edge-Response-Time: 131

{"desc":"default text from ResponseContentFilter"}%

So my guess is that they might be explicitly banning us from using this old endpoint and the binding might have to be rewritten.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/icloud-binding-communication-error/122131/377

[JensHoRi]

It’s not a binding issue, apple not in line with http specs at the moment

[RalphSester]

It’s not a binding issue, apple not in line with http specs at the moment

Source ?

[JensHoRi]

Al lot of research 6 month ago when same error was shown. You can look in the community too

[digitaldan]

@digitaldan: Do you have some ideas?

err, i don't use this binding and don't think i have contributed to it so i'm not sure how to help.

( i sometimes mix up threads about this binding and the openHAB Cloud binding since they sound similar in name, maybe thats the case here?)

[TDI4]

My solution was to install Home Assistant in my docker, installed iCloud integration, and established communication between the two systems via MQTT, and automations in HA. For example if me, as a person changing location, HA sends ON/OFF to my atHome, atWork switches. Or when my battery level changes, HA sends the value, and OH rule can warn me to charge my phone/watch.

It was urgent to solve the issue, because lot of my rules depending on if somebody is at home, or not. You can do that in less than 2 hours.

[JensHoRi]

OwnTracks with mqtt works fine too.

[BigGeorgeTx]

I am using OwnTracks with the GPSTracker Binding, which works well, perhaps better with my and my wife’s iPhone. Better because you can set a geofence and get real-time presence versus the potential 5 minute or more delay with iCloud.

But I use iCloud for devices like my AppleWatch that do not, at least as far as I know, support OwnTracks.

[mikthemonster]

Does anyone know where to find information about the "official" way to connect to iCloud, if we should re-write the binding?

[peteraquino]

Might be able to use the picklepete's pyicloud source code as a guide.

[mikthemonster]

I looked at picklepete's pyicloud and got it running on my Windows machine in Visual Studio Code. I can trigger "Find my iPhone" and get location etc. It is actually not very complicated.

I think it is "easy" to migrate the authentication principles to the the existing OpenHAB iCloud binding, but I dont have any clue about how to make or test an OpenHAB binding. I can probably make a Postman Collection showing all the principles and endpoints, but if some one could make the changes in the binding-code, then I would like to help making the changes - we could work together over Teams or something.

[mikthemonster]

We could also pay someone from freelancer to do it....

[maihacke]

If you could post the required HTTP request (inlcuding headers) here, I could have a look and try to create a PR + patched jar. The current implementation doesn't look too complicated.

[mikthemonster]

Hi Maihacke.

First Endpoint looks something like this (replace all the xxx with your own values):

HttpResponse<String> response = Unirest.post("https://idmsa.apple.com/appleauth/auth/signin?isRememberMeEnabled=true")
  .header("X-Apple-OAuth-Client-Id", "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d")
  .header("X-Apple-OAuth-Client-Type", "firstPartyAuth")
  .header("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com")
  .header("X-Apple-OAuth-Require-Grant-Code", "true")
  .header("X-Apple-OAuth-Response-Mode", "web_message")
  .header("X-Apple-OAuth-Response-Type", "code")
  .header("X-Apple-OAuth-State", "auth-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx") //Generate code - I think it can anything
  .header("X-Apple-Widget-Key", "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d")
  .header("Content-Type", "application/json")
  .body("{\r\n\"accountName\":\"[email protected]\",\r\n\"password\":\"xxxxxxxxxx\",\r\n\"rememberMe\":\"True\",\r\n\"trustTokens\": []\r\n}")
  .asString();

[maihacke]

Ok, what about the clientId. According to the docs it is something we need to acquire from WWDC:

https://developer.apple.com/documentation/sign_in_with_apple/clientconfigi/3230948-clientid

Is it okay to use the one from pyicloud?

[mikthemonster]

Hi Maihacke.

The ClientId from pyicloud is created using a function "uuid1" in python. It says: "uuid1(), uuid3(), uuid4(), uuid5() for generating version 1, 3, 4, and 5 UUIDs as specified in RFC 4122."

Unfortunately Apple has blocked sending more authenticationcodes to my iPhone at the moment - i cant test if other uuid's are working...

[mikthemonster]

Maihacke -> If you like we could make a teams meeting and I can show you the pyicloud in action. Then I guess it is easier for you to understand what it takes to implement in OH.

[maihacke]

The clientId is hardcoded here: https://github.com/picklepete/pyicloud/blob/332cc9fa767862480c27253233c2cfdf9f2ea0d9/pyicloud/base.py#L392

The state is generated, which is compliant with the docs. Regarding meeting, I'm currently very busy... Hope I can contribute here something nevertheless.

[mikthemonster]

Ok, there are two different client id's I was talking about client id in line 398 from your link above.

[maihacke]

I was able to send the mentioned request from within the binding. But this triggers a popup for 2FA on my local mac. I had a look on the pyicloud code and my impression is, that there is a lot more to do to handle 2FA correctly...

[mikthemonster]

Yes, im working on the next requests - It takes some time because Apple will only send a few codes to my phone pr. day, and then I have to wait.

[mikthemonster]

Again, if we could look at it together it would be easier I think.

[mikthemonster]

I finally got it working and made a Postman Collection:

https://www.getpostman.com/collections/407d255ecc18cdc8261c

1. Open the collection "APPLE ICLOUD"

2. Click on "Variables"

3. Enter your Email, pasword and a value for "X-Apple-OAuth-State". I think any value works - if not use something like auth-34792a3-3333-3333-3633-3377333333

4. Open and run "Auth_1_UN/PW". Some response values are stored in "APPLE ICLOUD" -> "Variables". And you shoukd get a code to your devices.

5. Open and run "Auth_2_ListServices".

6. Open "Auth_3_EnterToken" -> Open "Body" and enter the code and Send

--Now you are logged on and can do whatever To test Find My Iphone:

7. Open "FindMyIphone_1_RefreshClient". Copy your base url from "Auth_2_ListServices" if it differs from mine, and click Send. This should return a list of all your apple devises.
8. Open "FindMyIphone_2_PlaySound" and enter "ID FROM FindMyIphone_1_RefreshClient" for the device you want to find.

Thats it.

[maihacke]

From the community forum I found that somebody already create a java port from pyicloud. That would be possible of great help, if it works ;-): https://github.com/tmyroadctfig/icloud4j I'm not sure if it supports 2FA and how the input of the code could be integrated in the openhab flow, since there is no way of user interaction when starting the binding.