Sending spans, metrics and logs in one single request

[scheler]

What are you trying to achieve? For client-side telemetry (mobile apps, browser apps), it's prohibitive to send spans, metrics and logs separately on different connections. I want to explore ways to send them all together on a single connection/request. Creating this issue to invite ideas and comments.

What did you expect to see? One way to achieve this is to have each exporter delegate further to a common exporter which then uses a single gRPC connection to multiplex the data of different types.

It's not clear yet if client-side telemetry will use gRPC, so for HTTP case we will need to define a new Service endpoint in the protocol to accept a combination message.

Additional context.

Section 8.1.4 of HTTP/1.1 spec says the following: https://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html "A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy"

[jsuereth]

AFAIK - Today none of our SDKs / OTLP Exporters actually allow you to batch (consistently) across all signals. Indeed, architecturally there's a lot of friction to accomplish this (especially across the metrics + log/trace divide).

Would you be willing to make a proposal that includes:

• What this would look like from an SDK standpoint for mobile apps, browser apps, "constrained" environments?
  • e.g. what are the implication of attempting to batch all telemetry together.
  • How would you bundle / control the size of any given batch, practically, between the SDK components?
  • Do you simply mean there should be ONE http connection that you keep alive and dump telemetry down? (I think this may be more practical, and there was a previous discussion on it I'm failing to find)
• What you'd like to see this look like on the OTEL collector side (i.e. the OTLP gRPC service definitions).

[t2t2]

Adding as a reason to minimize the amount of requests, in web browsers there's a limit of how many simultaneous requests/connections can be done to one server/hostname (eg. ingest.example.com) which is generally around 6. This is why a common web performance improvement suggestion is to minimize the amount of requests

This would be especially problematic on slow (high ping) connections, if we take an example where sending data takes 3s while data is exported every second (generally want to send data frequently to minimise data loss on sudden user side force close of an app and to keep data amount low when user is leaving the page where there's limits on how much data is sent):

[tigrannajaryan]

If we decide that we want this then from OTLP perspective this probably can be added as a negotiated new capability, so that it is only used if the server indicates that it is ready to accept the mixed payload. This is important to ensure interoperability of clients and servers. The negotiation can happen via http headers.

[Aneurysm9]

How would this negotiation work and what improvement does it offer? It seems that it would add round trips required to determine whether the capability is available, mitigating the potential benefit.

Do you simply mean there should be ONE http connection that you keep alive and dump telemetry down? (I think this may be more practical, and there was a previous discussion on it I'm failing to find)

This seems to me to be the way to get the desired benefit, ideally an HTTP/2 connection that can perform request multiplexing. I believe the gRPC exporters should already support this if they can support user-provided connections to enable sharing across signal types without requiring the individual signal exporters to be aware of each other.

There are also patterns that have been developed to (partially) work around the concurrent connection limitation mentioned, which can be as simple as using a distinct service name per signal, i.e. logs.example.com and metrics.example.com, allowing for each signal to maximize concurrent connections and to avoid one slow signal degrading the performance of other signals.

Section 8.1.4 of HTTP/1.1 spec says the following: https://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html "A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy"

This is outdated advice. RFC 2616 was obsoleted by RFC 7230, which itself has been obsoleted by RFC 9110 and RFC 9112. The current STD 99, embodied in RFC 9112, has this to say:

Previous revisions of HTTP gave a specific number of connections as a ceiling, but this was found to be impractical for many applications. As a result, this specification does not mandate a particular maximum number of connections but, instead, encourages clients to be conservative when opening multiple connections.

I think we need to be very careful here about commingling performance concerns from a particular domain, that have a variety of patterns to address and tradeoffs to be made, with the design of our data model and service definitions.

[tigrannajaryan]

How would this negotiation work and what improvement does it offer? It seems that it would add round trips required to determine whether the capability is available, mitigating the potential benefit.

I want to make sure we all understand that it cannot be simply bolted on as another request type that some parties may not support and which offloads the interoperability responsibility to the end user. Any new enhancement to OTLP must be done in a way that makes it our responsibility to ensure interoperability of all existing OTLP implementations. Negotiation with graceful degradation is one of the possible ways to achieve it, but we can certainly consider others if and when we decide the enhancement is necessary.