[Service graph] Add prefix to differentiate client and sever attributes

[mapno]

In the service graph processor, when adding custom dimensions to metrics, values can be overridden if parent and children span have the same attribute but with different values. The attributes of the last span to be processed are the ones selected for the metrics. This results in inconsistent behaviour.

We should a prefix to the label metrics to differentiate between values coming from the parent span and the children span.

[yvrhdn]

Just a note: this will probably be a breaking change. If people were relying on specific Prometheus attributes these might change now. I think this is fine, we just have to document it.

[domasx2]

this has become important for app o11 efforts, we want to identify service by both service name and service namespace, so should be able to have namespace labels for both client and server

[zalegrala]

Do we have an update as to the next steps here? I see the PR is up, but do we have someone who is assessing the impact to existing customers and planning to communicate that impact?

[gouthamve]

So while we do have a solution for migration of customers. However, this will also break functionality in the service graph view. As discussed offline, the way to proceed here is a config option similar to enable_client_server_prefix: true. This would mean its opt-in and no existing users / functionality is affected.

And then in the future deal with how to align the service graph view to use these new metrics.