Add mobile device mapping ingest processor to apm ingest pipeline

[bryce-b]

Rescoping:

When https://github.com/elastic/elasticsearch/issues/88865 is implemented, add an ingest processor to the apm default ingest pipeline for mapping device.* related fields via using the new device mapping processor from ES.

---

Outdated:

Device model id captured on iOS devices come in a form that wouldn't be recognizable to users ( e.g.: iPhone8,2 is an iPhone 6S Plus, and iPhone8,4 is a GSE iPhone SE). We can map these values on the device, but when a new device comes out it would require an agent update to accommodate new devices.

Here's an example of device model ids and their corresponding device names : https://gist.github.com/adamawolf/3048717

[axw]

Where do we get this ID? Is this something that we can get from the User-Agent?

We can map these values on the device, but when a new device comes out it would require an agent update to accommodate new devices.

Does this happen frequently enough that that would be a problem?

[bryce-b]

I hadn't thought of the user agent! I'll check to see if that has the data we need.

Does this happen frequently enough that that would be a problem?

Apple releases a couple of new devices almost every year. If we depended only on mapping in the SDK, then when a new device comes out it wouldn't simply be a PR we need to make, but customers would have to update their app with the new SDK, then their customers would have to upgrade--a pretty slow turnaround.

I've attached a screen-cap of the UI featuring the device model

[simitt]

@bryce-b we are currently targeting 7.15 for this, please let us know if this is more urgent.

[bryce-b]

👍

[simitt]

@axw @bryce-b are we still blocked on this?

[axw]

I hadn't thought of the user agent! I'll check to see if that has the data we need.

@bryce-b did you ever find out an answer to this?

[axw]

Adding a little more flavour to my question above: I'm thinking that this functionality would be better placed in an Elasticsearch ingest processor. That would make it available to others, e.g. so it could be used in log parsing.

If we can identify the device ID via user-agent (I think probably not from what I've seen?), then perhaps the user_agent processor could be extended like in https://github.com/elastic/elasticsearch/pull/69322. Otherwise perhaps a new processor, or a canned enrich index/processor.

[bryce-b]

@axw You're correct, the user agent carries info like iOS version, but not model information. I'm still not familiar with all the ins and outs of the stack, so if having this functionality in an ingest processor make more sense, then I'll take your word for it! I don't know if it makes sense to add this to the user agent processor, however, if it's too difficult to get at the model information in the payload for one of these processors, the info could easily be added as a header

[simitt]

@bryce-b can you let us know if this becomes urgent? Pushing that further back for now.

[simitt]

@bryce-b can you please provide a current mapping so we can take a look?

[bryce-b]

@simitt We are just discussing this in the mobile sync, and we have decided to add this as an ingest processor to elastic search. I'll link the issue once it's created.

[simitt]

Thanks for the heads up @bryce-b. Since the ingest processors live in the apm-server repo (https://github.com/elastic/apm-server/tree/main/apmpackage/apm/data_stream) - I suggest we just repurpose this issue (this way we also keep the history of the decision on how to move forward).

[simitt]

The revisited goal is to add the mapping directly to ES, as an ingest processor (https://github.com/elastic/apm/issues/666, also see https://github.com/elastic/ecs/pull/2013).

Rescoping this issue to add the processor to the relevant apm pipelines. This is unlikely to happen during 8.5, rescheduling as 8.6-candidate.

[bryce-b]

Here's the ES ticket I created https://github.com/elastic/elasticsearch/issues/88865