Redis/MySQL/... remote services don't differentiate hosts

[MartinKolarik]

Describe the bug

We have multiple redis servers in use - some services use their own redis available on localhost, some access a shared remote instance by its IP. The Service Map view in APM doesn't differentiate between these hosts and shows all of them as a single "redis" node used by all services. The same happens e.g., with MySQL, while HTTP remote services are correctly split by their hostname.

I'm not entirely sure if this is just the node agent not collecting all the necessary info, or a broader issue with the services spec, so feel free to move this issue to the right place if it's not directly the agent's fault.

Environment (please complete the following information)

• OS: Linux
• Node.js version: v14.17.1
• APM Server version: v7.15.1
• Agent version: 3.23.0

How are you starting the agent? (please tick one of the boxes)

• [x] Calling agent.start() directly (e.g. require('elastic-apm-node').start(...))
• [ ] Requiring elastic-apm-node/start from within the source code
• [ ] Starting node with -r elastic-apm-node/start

Additional context

• package.json dependencies:
  Click to expand

  Using the redis module in some cases and ioredis in others, in case it's relevant.

[trentm]

I'm not entirely sure if this is just the node agent not collecting all the necessary info, or a broader issue with the services spec

@MartinKolarik Thanks! I think more the latter. The service map is using the span.context.destination.service.resource value, which for redis is spec'd to be just "redis" -- i.e. no differentiating Redis instance info.

I am asking around internally first and will likely either start an spec discussion issue and link here, or transfer this one to be that.

[trentm]

Though it says "for SQL Databases", https://github.com/elastic/apm/issues/473 is basically the same issue here: increasing the granularity on the data collected by APM agents for backends (often DBs) and using that data in the APM UI (Service Map and Dependencies).

My inclination is to close this issue and add a note to https://github.com/elastic/apm/issues/473. Currently that issue is the best one for you to follow our plans and work on this. There is active internal discussion for improving this, but no current public timeline yet.

In the short term, here is a possible inadvisable :) hack: add a span filter that cheats by munging the span.context.destination.service.resource field before the agent sends it to APM Server. A example span object looks like this:

    {
        "span": {
            "name": "HKEYS",
            "type": "cache",
            "id": "f2405e6f2b774457",
            "transaction_id": "ea1190cc6e8c844a",
            "parent_id": "e6e6f3ca6afceadc",
            "trace_id": "71739d85857e8b0d5c57a11314770952",
            "subtype": "redis",
            "action": null,
            "timestamp": 1636138717125220,
            "duration": 2.246,
            "context": {
                "destination": {
                    "service": {
                        "name": "redis",
                        "resource": "redis",
                        "type": "cache"
                    },
                    "address": "127.0.0.1",
                    "port": 6379
                }
            },
            "sync": true,
            "outcome": "success",
            "sample_rate": 1
        }
    }

You could have a span filter that pulls out that "address" field and appends it to the "resource" value. Be warned that this'll break future efforts by Elastic APM to automatically do this generally -- and possibly more "safely" by watching out for cardinality issues in the "resource" value.

[MartinKolarik]

Agree, https://github.com/elastic/apm/issues/473 is the same type of problem, it just needs to be addressed more broadly.

[trentm]

This might be answered when https://github.com/elastic/apm-agent-nodejs/issues/2621 is done.