☢☢ The FmF (Find-my-Friends) Tracking Method is not working due to a change Apple made

[gcobb321]

Apple has made a change to the data returned from iCloud Location Services. The url links to that would return the FmF data are no longer returning any FmF data. It looks a lot like the FamShr data but I can not be sure if it is the same data or only part of it.

The location for the friends is still being reported correctly in the FindMy app but not being sent with iCloud3 requests it.

I will be investigating the issue and see what can be done. In the meantime, the FamShr tracking method is working fine.

[gcobb321]

I’m not seeing any problems with famshr

[miraixyz]

nope...Famshr is not working either

[gcobb321]

FamShr it’s working fine in iCloud3 v3.

[szabomarcihu]

Find-my-Friends it works again. iCloud3 v2 :)

[miraixyz]

Sorry. I forgot to uninstall the previous version. Works perfectly now...thanks

[SaturnusDJ]

Find-my-Friends it works again. iCloud3 v2 :)

Are you sure? FmF? v2?

Here: Requesting iOSApp Location #2 > LastLocTime-16:37:11 (2 hrs ago), LastRequested-18:37:25 (25 sec ago) > Failed to send message

2023-05-15 18:33:44.183 ERROR (SyncWorker_12) [custom_components.icloud3.pyicloud_ic3]  (Status Code 501)
2023-05-15 18:33:44.185 ERROR (SyncWorker_12) [custom_components.icloud3.device_tracker]  (Status Code 501)
Traceback (most recent call last):
  File "/config/custom_components/icloud3/device_tracker.py", line 6776, in _setup_tracked_devices_for_fmf
    self._log_rawdata("iCloud FmF Raw Data - (api_friends.data)", api_friends.data)
  File "/config/custom_components/icloud3/pyicloud_ic3.py", line 1079, in data
    self.refresh_client()
  File "/config/custom_components/icloud3/pyicloud_ic3.py", line 1006, in refresh_client
    req = self.session.post(self._friend_endpoint, data=mock_payload, params=params)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 635, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/config/custom_components/icloud3/pyicloud_ic3.py", line 179, in request
    self._raise_error(response.status_code, response.reason)
  File "/config/custom_components/icloud3/pyicloud_ic3.py", line 239, in _raise_error
    raise api_error
custom_components.icloud3.pyicloud_ic3.PyiCloudAPIResponseException:  (Status Code 501)

[EvanGrote]

I'm seeing the same error as @SaturnusDJ on every device I track that is configured as iosapp_installed: false.

Checking my HA history, all of my non-iosapp_installed devices stopped returning updated battery data a little after 8pm (EST) on May 10th

[gcobb321]

An error 501 is the Service (FmF) is not Available. Battery info is only available on FamShr and iOS App devices. If you are not using iOS App and FamShr is having an issue, no battery info. iC3 is probably not checkin to see if the iOS App is installed on a device before trying to send a message.

I am assuming you are having the problem with v2, not v3.

[EvanGrote]

I am assuming you are having the problem with v2, not v3.

Ah! I hadn't put together that you have a v3 in the wild. I think my mind was reading "iCloud3 v3" as just "iCloud3".

I have now migrated to the v3 beta and everything seems to be working great again. Thank you for your work on this integration!

[gcobb321]

Glad to have you trying it

[gcobb321]

How to solve this problem.

1. Upgrade to iCloud3 v3. It solves this issue and adds a lot of new features.. iCloud3 v3 is almost ready to release when some of the docs have caught up with the code. It is a full Integration with all configuration done using configuration screens. Go here for more info and installation instructions. It’s still in beta because I’m adding a few tweaks and the docs are not complete.

• Important Links
  • Installing using HACS - iCloud3 is not available on the HACS base. You have set it up as a custom repository. See here for instructions on how to do that and then install it. If you just go to HACS and do the installation, you will be installing v2.4.7, not v3.
  • Download and Install - Go to the iCloud3 Releases page here. Scroll down past the notes, then select Assets, then select icloud3.zip. Download the zip file and unzip it into the /config/custom_components/icloud3 directory. Then restart HA.
  • Migrating from v2.4._ - See here for instructions on migrating from from an older version.
  • iCloud3 v3 Documentation - It is still in process, does not include the latest updates to iCloud3 and can be found here

or

2. Edit the /icloud3/device_tracker.py file. Comment out line 1214 by putting an '#' at the start of it so it looks like

1213                if self.api:
1214                    #self._setup_tracked_devices_for_fmf()
1215                    self._setup_tracked_devices_for_famshr()
1216                    self._tune_tracked_devices_fmf_famshr()

Save the file and restart HA.

[tomeralgai]

1. Upgrade to iCloud3 v3. It solves this issue and adds a lot of new features..

I just installed Icloud3 V3 (c3.0.0b19.1) with the hopes of being able to see the data from the FmF method but all I see is this: Did I do something wrong?

[gcobb321]

No. FmF is not available from Apple. I'll be looking for a solution after iC3 v3 has gone live.

[sossienl]

No. FmF is not available from Apple. I'll be looking for a solution after iC3 v3 has gone live.

I did some sniffing in the FMF corner and found the following endpoints. Not sure if this is helpful but it also cannot hurt to share this! I removed data that seems to be personal.

There is a INIT call that shows the following URL:

NUMBER1: 10 digit number, for me starting with 1018...... STRING1: 8 digits, followed by a "-" ending with 16 characters (numbers and letters).

https://p39-fmfmobile.icloud.com/fmipservice/friends/NUMBER1/STRING1/first/initClient

Response (anonymised and not full response): { "modelVersion": "1", "dataContext": { }, "futureFollowing": [], "features": { "addEventFriends": true, "airDropOffer": true, "viewFollowers": true, "removeFriends": true, "removeDevice": true, "notifyOthers": true, "canChangeGeoFenceAlerts": true, "actOnRequest": true, "actOnEventRequests": true, "createEvent": true, "viewFriends": true, "changeMyLabel": true, "deleteEvent": true, "changeMeDevice": true, "changeHideLocation": true, "offerLocation": true, "editEvent": true, "viewEvents": true, "leaveEvent": true, "discoverContacts": true, "changeAllowFriendRequests": true, "removeEventFriends": true, "removeFollowers": true, "sendRequest": true, "notifyMe": true }, "futureFollowers": [], "friendFencesOthersSet": [], "events": [], "futureEvents": [], "pendingOffers": [], "labelledLocations": [{ "latitude": <<REMOVED>>, "labelType": null, "label": "_$!<work>!$_", "id": "<<REMOVED>>", "updateTs": 1382532362935, "userId": null, "longitude": <<REMOVED> }, { "latitude": <<REMOVED>>, "labelType": null, "label": "_$!<home>!$_", "id": "<<REMOVED>>", "updateTs": 1444753397316, "userId": null, "longitude": <<REMOVED>> }], "devices": [{ "deviceIsFencable": true, "name": "iPhone", "idsDeviceId": "<<REMOVED>>", "id": "<<REMOVED>>", "autoMeCapable": false }, { "deviceIsFencable": true, "name": "iPad", "idsDeviceId": "<<REMOVED>>", "id": "<<REMOVED>>", "autoMeCapable": false }], "fetchStatus": "200", "myFencesISet": [], "contactDetails": [{ "emails": ["<<REMOVED>>", "<<REMOVED>>"], "firstName": "Jane", "photoUrl": "PHOTO;TYPE=JPEG;<<REMOVED>>", "lastName": "Doe", "contactId": "<<REMOVED>>", "phones": ["<<REMOVED>>"], "middleName": "", "id": "<<REMOVED>>" }, { "emails": ["<<REMOVED>>", "<<REMOVED>>"], "firstName": "John", "photoUrl": "PHOTO;X-ABCROP-RECTANGLE=ABClipRect_1&<<REMOVED>>", "lastName": "Doe", "contactId": "<<REMOVED>>", "phones": ["<<REMOVED>>", "<<REMOVED>>"], "middleName": "", "id": "<<REMOVED>>" }], "friendFencesISet": [], "prefs": { "allowFriendRequests": "Yes", "fenceNotification": "EVERYONE", "hideLocation": "No", "primaryEmail": "<<REMOVED>>", "shouldReceiveEmails": "Yes", "favorites": [{ "id": "<<REMOVED>>", "order": 0 }] },

I see multiple requests towards url:

https://p39-fmfmobile.icloud.com/fmipservice/friends/NUMBER1/STRING1/minCallback/selFriend/refreshClient

Response:

{ "dataContext": { "11": <<REMOVED>, "22": <<REMOVED>, "12": <<REMOVED>, "13": <<REMOVED>, .... }, "locateInProgress": [{ "id": "<<REMOVED>~~", "status": "none" }, { "id": "<<REMOVED>", "status": "none" }], "serverContext": { "minCallbackIntervalInMS": 5000, "res": null, "inaccuracyRadiusThreshold": 200, "clientId": "<<REMOVED>", "showAirDropImportViewOniCloudAlert": true, "authToken": "<<REMOVED>", "maxCallbackIntervalInMS": 15000, "fallbackToLegacyIntervalInSec": 10.001, "reverseGeocodingThrottleDistance": 100.0, "prsId": <<REMOVED>>, "callbackTimeoutIntervalInMS": 0, "heartbeatIntervalInSec": 543600, "liveTimeoutThreshold": 30.001, "liveAnimationInterval": 2.001, "reverseGeocodingThrottle": 30.001, "transientDataContext": { "0": <<REMOVED>, "1": <<REMOVED>, "2": <<REMOVED>, "3": <<REMOVED>, "4": <<REMOVED> }, "sendMyLocation": true, "notificationToken": null, "iterationNumber": 3 }, "modelVersion": "1", "fetchStatus": "200" }

And in the 3rd response to this url I see this:

{ "serverContext": { "minCallbackIntervalInMS": 5000, "res": null, "clientId": "<<REMOVED>>", "inaccuracyRadiusThreshold": 200, "showAirDropImportViewOniCloudAlert": true, "authToken": "<<REMOVED>>", "maxCallbackIntervalInMS": 15000, "fallbackToLegacyIntervalInSec": 10.001, "reverseGeocodingThrottleDistance": 100.0, "prsId": <<REMOVED>>, "callbackTimeoutIntervalInMS": 0, "heartbeatIntervalInSec": 543600, "liveTimeoutThreshold": 30.001, "transientDataContext": { "0": <<REMOVED>>, "1": <<REMOVED>>, "2": <<REMOVED>>, "3": <<REMOVED>>, "4": <<REMOVED>> }, "liveAnimationInterval": 2.001, "reverseGeocodingThrottle": 30.001, "sendMyLocation": true, "iterationNumber": 4, "notificationToken": null }, "modelVersion": "1", "fetchStatus": "200", "dataContext": { "11": <<REMOVED>>, "22": <<REMOVED>>, "12": <<REMOVED>>, "13": <<REMOVED>>, ... }, "locations": [{ "locationStatus": null, "location": { "altitude": 0.0, "isInaccurate": false, "address": { "locality": "<<REMOVED>>", "formattedAddressLines": ["<<REMOVED>>", "<<REMOVED>>", "<<REMOVED>>"], "country": "<<REMOVED>>", "administrativeArea": "<<REMOVED>>", "countryCode": "<<REMOVED>>" }, "locSource": null, "secureLocation": null, "secureLocationTs": 0, "latitude": <<REMOVED>>, "floorLevel": 0, "horizontalAccuracy": <<REMOVED>>, "labels": [{ "type": "premises", "label": "<<REMOVED>>", "info": "<<REMOVED>>" }], "tempLangForAddrAndPremises": null, "verticalAccuracy": 0.0, "batteryStatus": null, "locationId": "<<REMOVED>>", "locationTimestamp": 0, "locationMode": null, "longitude": <<REMOVED>>, "timestamp": <<REMOVED>> }, "id": "<<REMOVED>>~~", "status": null }], "locateInProgress": [{ "id": "<<REMOVED>>~~", "status": "none" }] }

[gcobb321]

Very interesting. I’d like to look at the code you found in the FMF corner to get a better understanding of how it is configured and used. It would also be helpful to see the code you used. I understand the username/password security issues so use [email protected] for communication.

Thanks.

[dominicstelljes]

Is it working again or is it still not possible? :)

[gcobb321]

@dominicstelljes I think it is dead forever.

[dominicstelljes]

Too bad, thanks for the quick reply!

[nbetcher]

Just curious how you went from this:

Very interesting. I’d like to look at the code you found in the FMF corner to get a better understanding of how it is configured and used.

to this:

I think it is dead forever.

Are the API responses that @sossienl posted only for the native app (and thus more heavily regulated)? Or is there some other technical problem? Obviously Find My Friends didn't go away as a product offering, and nothing is truly magical in this dimension, so I'm just curious how difficult Apple is making it for you. :-)

[gcobb321]

@nbetcher The actual source came from pyicloud at the picklepete GitHub repository with support for Find my Friends at z. Zelnick’s repository. I have customized it for icloud3 to handle the data returned from iCloud differently but the http web iCloud access is from those sources. The problem is that the fmf url returned location info from iCloud and then became unavailable, returning a 404 or another error. It’s been a while since I looked at it. After trying various urls and methods of getting the data, and since the original url is not supported or not documented by Apple as a supported url, it seemed like it was dead and fmf was no longer available.

The code and other info is still in my version of pyicloud and can be found in the gcobb321 icloud3_v3 GitHub repository in the custom_components/icloud3/support/pyicloud_ic3.py file.

If you have any interest trying to reimplement this, let me know.

Gary

[nbetcher]

My interest is unfortunately fairly minimal given my need, but it's not entirely out of the question.

My primary question is: since you can no longer use Find My to find your friends on the web version of iCloud (and only your own devices, it seems), I would need to reverse engineer their private API in the native app. I only have access to an older MacBook Pro running Sonoma, but from what I've read no one had bothered to hack MacOS (recently) to unpin Apple's SSL/TLS certificates from their own apps, thus making sniffing the traffic impossible.

While I realize this isn't the forum to be discussing these sorts of things, I'd appreciate it if anyone with such unpinning advice (or other avenues of "attack") please message me privately here.

Until then, my interest is limited.

Thanks, 一Nick

[bitraker]

Is there still no way to get FMF working?

[gcobb321]

None. And it will probably never work again. I am in the process of adding support for multiple Apple accounts to provide another way of tracking devices that are not in your family share list