Define a way to specify more than one Resource in telemetry data

[tigrannajaryan]

Currently it is possible to refer to a Resource in spans and metrics. The Resource indicates where the signal was generated.

However, we may need a way to reference the second (or third, etc) resources.

For example an application A (Resource 1) may report a metric such as "request count" and it may be the number of requests that application A made to application B (Resource 2). Thus, this requires that the metric references multiple resources.

See discussion context here: https://github.com/open-telemetry/oteps/pull/97#discussion_r418747530

[Oberon00]

See also issue on HTTP app reporting (or per span service resource depending how you look at it): https://github.com/open-telemetry/opentelemetry-specification/issues/335 Also related: https://github.com/open-telemetry/opentelemetry-specification/issues/274 "Allow resources as span attributes"

[jmacd]

Related content here: https://github.com/open-telemetry/oteps/issues/78

To support resources as an API-level concept, as in OTEP 78, requires protocol support for "more than one Resource" as you say.

[tigrannajaryan]

One new use case for this is eBPF. Network monitoring collector needs to record telemetry about 2 entities communicating, each entity can be one of the Resource types that we record using semantic conventions (e.g. network traffic between 2 pods). It is currently not possible to do since there can be only one set of semantic conventions per type of entity recorded in the Resource. Even if we could record 2 sets of conventions in the Resource it doesn't seem to the right approach, since the Resource is supposed to be the indicating the origin of the telemetry.

[duxing]

@tigrannajaryan totally agree with what you said. any application that has network IO to other entities (service, cache, db, etc) should really reflect the traces with 2 entities per communication. one for the client (self, source of network io) and one for the client's observation of server entity (destination of network io).

[grepory]

This has come up a lot, and I continue to be a bit of a purist for Resources that a Resource is something that is the "source" of telemetry.

A metric from ServiceA, RequestCount, is--in an of itself--the number of external requests that ServiceA may make.

Destination is an attribute/label/dimension of the RequestCount metric.

If ServiceB and ServiceC are Resources on the ResourceMetrics, how do you differentiate between multiple RequestCount metrics within a ResourceMetrics blocks as to which count is for ServiceB and which is for ServiceC? Do you duplicate the data?

Even in the case of network flows, you may be observing bidirectional communication, but I think the existing ResourceSpan data model doesn't overly complicate things. You can just as easily have Resource be the observer of the flow and then record source and destination within your trace spans.

We're also more likely to only have partial resources for external entities for attributes that are determined at runtime. E.g. service vs. service instance.