Batch Reports

[paulb-firebolt]

Previously we used to fetch batches of device (64 at a time), but something changed, and we started getting only reports for the first device in the list.

I'm going to start looking at this, but wondered if those with an understanding of the Apple API thought it's reasonable.

The Code in 0.9.4 (account.py)

"fetch": [
    {
        "ownedDeviceIds": [],
        "keyType": 1,
        "startDate": start_ts,
        "startDateSecondary": start_ts,
        "endDate": end_ts,
        "primaryIds": device_keys[0],
        "secondaryIds": device_keys[1],
    }
    for device_keys in devices  # <-- Still creates multiple fetch objects!
],

This list comprehension creates one fetch object per device batch, which is the problematic pattern.

Current Problematic Behaviour

The current FindMy.py implementation creates multiple fetch objects when fetching reports for multiple devices:

{
  "fetch": [
    {
      "keyType": 1,
      "primaryIds": ["device1_hashed_key"]
    },
    {
      "keyType": 1,
      "primaryIds": ["device2_hashed_key"]
    },
    {
      "keyType": 1,
      "primaryIds": ["device3_hashed_key"]
    }
  ]
}

Result: Only device1 reports are returned; device2 and device3 are ignored.

Workaround:

Flatten all device keys into a single fetch object:

{
  "fetch": [
    {
      "keyType": 1,
      "primaryIds": [
        "device1_hashed_key",
        "device2_hashed_key",
        "device3_hashed_key"
      ]
    }
  ]
}

Code Location: findmy/reports/account.py in fetch_raw_reports() method.

Recommended Fix

# Instead of:
"fetch": [
    {"primaryIds": device_keys}
    for device_keys in devices
]

# Use:
all_keys = [key for batch in devices for key in batch]
"fetch": [
    {"primaryIds": all_keys}
]

This maintains the 256-key per-request limit while ensuring all devices are queried.

[AGaylord6]

Interesting solution (or API hack) proposed and implemented by chapoly: https://github.com/biemster/FindMy/issues/82#issuecomment-3458857366. Seems to related to what you're suggesting, I haven't tested though

[malmeloo]

Sorry for taking a while, my notifications are a bit wonky it seems. chapoly's description is how FindMy.py currently does it, and I haven't noticed any issues so far. Putting all keys into a single 'device' in the request (such as what @paulb-firebolt describes) makes it so that Apple will only return the last 20 reports for all devices in the request, which isn't really desirable either.

Is anyone else running into this issue? Maybe there's just a limit to the number of devices that can be provided in a single request? I don't really own 64 devices to test that myself though.