request_reports no longer downloading data from multiple keys files.

[swiss7]

As of yesterday, it seems that the request_reports script will no longer download data from all of the .keys file found in a folder. It seems that currently it only takes one of these keys files, with all of the others reporting as 'missing'. Yet the 'missing' one will download data if it is the only keys file in the folder.

Any ideas? Has something changed on the server side? Thanks!

[swiss7]

It also seems to be limiting the downloads to 20 reports? Strange change of behaviour.

(btw I am using version 1 of the script, but I don't think that this should make a difference)

[biemster]

Probably something changed on Apple's side, maybe only a single tag can be requested now? Report downloads were already limited to 20 when no startDate or endDate was provided (or something like that), it might be that that got changed as well. I'll have a look coming week, thanks for reporting!

[vasimv]

Yeah, two days ago i've got same problem - multiple keys request is limited to 20..40 reports per request and, usually, it all gets filled by one of the keys. Had to convert script to request keys one by one. May lose some data as one of beacons can get 20+ new reports in between requests, but at least it works.

[swiss7]

Thank you both for confirming. I had come to the same conclusion, but was really hoping that I had somehow missed something. Hopefully someone cleverer than me can figure out if it's just a syntax change and that we can still use something similar to the previous approach.

Vasimv can you share the block of code that loops through the keys files? Thanks!

[vasimv]

My dirty hack: onebyone.patch.txt

[lovelyelfpop]

@vasimv If I request keys one by one, too many requests may cause http status code 401, then Expecting value: line 1 column 1 (char 0)

[lovelyelfpop]

No wonder I can’t get the latest location for almost all the tags I made. This is because each request for multiple keys will only return 20 to 40 location reports, and these location reports may only belong to one of the keys.

[swiss7]

One thing that remains unclear—what determines whether the cloud sends back 20 versus 40 reports? I seem to only receive 20 (giving both a start and end date in the request).

edit: I think that this is something to do with not giving the correct startDate and endDate values in the json call, did folks do something different here?

edit2: I just found the earlier post about start and end times, so if this is generally not functioning, how then are people (e.g. @lovelyelfpop) receiving 40 reports? I'm always stuck on 20. Note that I'm using the previous version (without Anisette), so that could be the difference.

[josephBoonrawd]

I want too. Next step I create 60000 tag for system beertacker. But I don't know if it's possible.

[halex999]

Same problem for me as well. I use FindMy with traccar and did not get any position updates since friday.

[khongpt]

I've encountered a similar issue, where out of 13 tags, suddenly only 1 tag's information was returned. After several attempts, I found that Apple Find My seems to have some kind of limit—the number of records in each tag's report is around 20–23. Therefore, I recommend adjusting the process to fetch reports for each tag individually, sequentially retrieving the remaining tags

[lovelyelfpop]

I modified request_reports.py. I use async and parallel requests to save time. See: https://gist.github.com/lovelyelfpop/9733899e3793df3b74d1da57095c46ce

[olivluca]

Won't you get banned/throttled if you issue too many requests?

[lovelyelfpop]

@olivluca 4 requests at one time, not get banned yet.

[jl-jung]

@josephBoonrawd We are currently working successfully on a similar project. Would you be interested in connecting for an exchange of ideas?

[tobox]

Any news on this? Currently, I cannot get any results. Maybe apple has changed something on their side again?

The modified script of lovelyelfpop returns this:

Request failed: BasicAuth() tuple is required instead Request failed: BasicAuth() tuple is required instead 0 reports used. found: [] missing: ['VMC4O6', 'Token1', 'Token3', 'Token2']

[halex999]

The same result for me

[tobox]

Strange. I entered the password and 2FA once more and now it works again.

[josephBoonrawd]

@josephBoonrawd We are currently working successfully on a similar project. Would you be interested in connecting for an exchange of ideas?

@jl-jung Yes. I would be intersted.

[sonman]

When sending 9 keys in one request, I get back reports for all of them. Total ~170 reports in response. Maybe if you have ~200 keys in a request only one will receive reports ?

[aircable]

After some optimization runs, I found out, the longer the query the less number of reports the iCloud gives you. The best number was asking for 4 keys in one query, 4 queries at a time. Anybody else did those tests?

[IDkonnecT]

@aircable Clear how you put several keys in one query, but could you help understand how to formulate several queries at a time inside same post request ?

[malmeloo]

It could be that Apple is throttling the keys for all users of the API, including official clients. I've just taken a look at some network logs of my Sonoma VM, and it's using a completely different API endpoint: /findmyservice/v2/fetch instead of /acsnservice/fetch as documented in the OpenHaystack paper. This API appears to only return a single location report per device though, which is quite the downgrade from what we have now.

Also, I'd like to urge everyone not to spam the API as a 'solution' to the problem; that might urge Apple to actively try to block 3rd party uses of the API and could ruin the fun for the rest of us.

[biemster]

It could be that Apple is throttling the keys for all users of the API, including official clients. I've just taken a look at some network logs of my Sonoma VM, and it's using a completely different API endpoint: /findmyservice/v2/fetch instead of /acsnservice/fetch as documented in the OpenHaystack paper. This API appears to only return a single location report per device though, which is quite the downgrade from what we have now.

Also, I'd like to urge everyone not to spam the API as a 'solution' to the problem; that might urge Apple to actively try to block 3rd party uses of the API and could ruin the fun for the rest of us.

Awesome, thanks Mike! I've been trying to find the time to revive my VMs for some captures but did not manage yet. Can you give an example request json to the new endpoint, and did you implement this in your FindMy.py currently?

And indeed, definitely don't hammer the Apple endpoints because they are quite efficient in banning 3rd parties, remember Beeper?

[malmeloo]

I only managed to take a brief look at it, but I'll look into it again soon. From what I could tell it does appear to share more info about the actual device than before, including separation of the primary and secondary keys, and some device identifiers (?). I'm also not sure if they changed any authentication requirements or if it's just the same headers as before. Hopefully it won't be too difficult to migrate to the new API.

[malmeloo]

Okay, I was not entirely correct yesterday. It does still appear to return multiple reports, my airtag was just too young.

The new API endpoint is https://gateway.icloud.com/findmyservice/v2/fetch. I can confirm that authentication works exactly the same way, so basic auth using dsid + searchpartytoken and anisette headers should do the trick.

Here's an example request body:

{
   "clientContext":{
      "policy":"foregroundClient",
      "clientBundleIdentifier":"com.apple.findmy" // alternatively: "com.apple.icloud.searchpartyuseragent"
   },
   "fetch":[
      {
         "secondaryIds": [  // alt key name: "primaryIds"
           // key IDs, same as before
         ],
         "keyType":1,
         "endDate":1751846399999,
         "ownedDeviceIds": [
            // OF key IDs of account devices? just leave it empty
         ],
         "startDateSecondary":1751733308788,
         "startDate":1751733308788
      }
   ]
}

I observed two different clientBundleIdentifiers, it doesn't really appear to matter which one you use. They appear to separate key IDs into primary, secondary and owned device IDs now, but I'm not sure why. Fundamentally they're all the same anyway. Just don't put your custom tag's keys in the ownedDeviceIds, because those are listed separately in the response.

Example response:

{
   "configVersion":1,
   "acsnLocations":{
      "locationPayload":[
         {
            "locationInfo": [
               // encrypted payloads for the given key, format is stil exactly the same as far as i can tell
            ],
            "id": "" // hashed adv key
         }
      ],
      "configVersion":1,
      "statusCode":"200"
   },
   // this will be empty if you dont pass any ownerIds
   "ownedDeviceLocations":{
      "locationPayload":[
         {
            "locationInfo":[
               {
                  "locationTs":1751746897601,
                  "location": "", // just an encrypted report again
                  "fmt":0
               }
            ],
            "id": "" // hashed adv key
         }
      ],
      "configVersion":1,
      "statusCode":"200"
   }
}

So there's not a whole lot that changed really, it's mostly just a new req/resp format. We are however missing the description and publishedAt fields now, but afaik the former has always been empty and the latter was not too useful anyway. The new API does appear to be a lot faster in my tests.

I also don't think they really fixed the start/end timestamp issue, in my tests they still just return all reports anyway. I haven't tested the limit on how many keys it will return yet.

[swiss7]

Is it possible to get a full working example (e.g. of request_reports.py) using the new API endpoint? I'm not quite sure what I'm getting wrong when trying to implement it. Thanks a lot!

[biemster]

Is it possible to get a full working example (e.g. of request_reports.py) using the new API endpoint? I'm not quite sure what I'm getting wrong when trying to implement it. Thanks a lot!

It's implemented in malmeloo's https://github.com/malmeloo/FindMy.py/pull/144, but it would be even more useful if you could share the errors you're getting? (I did not find the time yet to implement this, but your work will greatly help with that!)

[swiss7]

Thanks for the quick reply!

It's implemented in malmeloo's https://github.com/malmeloo/FindMy.py/pull/144

Yep I figured, but I code so little in Python that I couldn't quite figure it out from that (sorry).

So I'm trying to implement the request body from the comment above: data = {"clientContext":{"policy":"foregroundClient","clientBundleIdentifier":"com.apple.findmy"},"fetch":[{"secondaryIds": [XXXXXXXXXXXXXXXXXXXXXXXXXXX],"keyType":1,"endDate":1751846399999,"ownedDeviceIds":[],"startDateSecondary":1751733308788,"startDate":1751733308788}]}

Then request this (per Monterey branch which I'm still using): response = requests.post('https://gateway.icloud.com/findmyservice/v2/fetch', headers=request_headers, data=data)

This gives me a correct response: response <Response [200]>

But then the content seems to be empty or wrong:

response.json() Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 971, in json return complexjson.loads(self.text, **kwargs) File "/usr/local/Cellar/[email protected]/3.9.19/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/init.py", line 346, in loads return _default_decoder.decode(s) File "/usr/local/Cellar/[email protected]/3.9.19/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/Cellar/[email protected]/3.9.19/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 975, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I know that this tag has data because it gives me 20 reports using my existing script.

Thanks again!

[biemster]

Can you print the response.text? (or maybe the whole response?)