Support for OTLP/UDP Span Exporter (Lambda+XRay use case)

[jj22ee]

Is your feature request related to a problem? Please describe.

X-Ray users who want to use OTel in AWS Lambda are recommended to use the OTel Lambda Layers that packages both the OTel Collector and OTel SDK components to instrument a Lambda Function. However, using OTel on Lambda may incur performance impact, partially due to using OTel Collector.

• https://github.com/open-telemetry/opentelemetry-lambda/issues/263 - issue summarizes the performance impact in Lambda

Describe the solution you'd like

The approach to reduce the performance impact in Lambda by OTel involves avoid running the Collector in Lambda. Lambda supports their own endpoint to receive OTLP/UDP spans and send them to AWS X-Ray. It is re-using the existing UDP endpoint that is currently used by the X-Ray tracing SDKs. It exists as an alternative to the Collector's OTLP Receiver.

Today, AWS Observability's auto-instrumentation is leveraging a UDP Exporter to export traces to Lambda's UDP endpoint to send traces to AWS X-Ray. I want to add this UDP Exporter to an OTel JS repository (either contrib or non-contrib).

Additional context

• At first glance, while the intent of this solution is for Lambda+XRay specificly, the exporter seems generic. Should this be named as a Lambda/XRay OTLP/UDP Exporter or just OTLP/UDP Exporter?
• There is no specification for this OTLP/UDP Exporter
• Issue moved from non-contrib otel-js repo

[jj22ee]

Copying @pichlermarc's question from previous issue:

What I'm wondering though is - IIUC the improvement would only take effect if the collector is removed. If that's the case and OTLP/UDP is not in the spec, what is the plan to keep this approach compatible with OTel's principle of being vendor-agnostic? Can you forward the Spans received via UDP somewhere else via OTLP/http or OTLP/gRPC without added cost? 🤔

You are right that this endpoint is specific to vending to X-Ray, not vendor-agnostic. This UDP Exporter solution is re-using the existing UDP endpoint at 127.0.0.1:2000 that is currently used by the X-Ray tracing SDKs. This means this performance impact mitigation solution is not compatible with the vendor-agnostic principal; the existing collector-in-Lambda solution is still needed to vend to non-XRay. I think AWS Lambda still needs a new solution from their end to mitigate performance in this scenario.

[pichlermarc]

@jj22ee hmm, I would not call it OTLP/UDP for the reason that it's not in the spec and it'd look like we're publishing an official OTLP/UDP exporter.

If it's an exporter that's only usable with AWS XRay then we should choose a name that makes it clear that that's what it is. Maybe @opentelemetry/exporter-aws-xray-lambda would be a suitable name? In any case, it's clear that this would be a so-called vendor-component and it would have to adhere to these rules.

A few follow-up questions:

• would this component be added to https://github.com/open-telemetry/opentelemetry-lambda?
  • if yes, please reference the issue here :)
• is there any benefit from hosting the new exporter here in this repo?
  • usually the only reason to have it here is if it's generic enough, or if it should go into any of the three metapackages (https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/metapackages)

If the answer is no to both of these questions, then it might be better to host it as a third-party component under the https://github.com/aws organization as that'd give you way more flexibility in terms of maintaining it since you'll be able to publish new versions whenever you want to and choose whatever tooling you need.

[jj22ee]

would this component be added to https://github.com/open-telemetry/opentelemetry-lambda?

Yes. Probably would've made more sense to start the UDP exporter conversation there :sweat_smile:, but here it is now: https://github.com/open-telemetry/opentelemetry-lambda/issues/1685

is there any benefit from hosting the new exporter here in this repo?

I was thinking it would make it easier for the OTel Lambda Layers to accept adding support for this UDPExporter component if it is hosted in an OTel repository. I was also thinking of the possibility of making it available through the OTEL_EXPORTER_OTLP_PROTOCOL env var (but I guess this is unlikely to be accepted given this UDPExporter is not in OTel spec?).

But speaking of the OTel Spec, I do wonder if anyone there has any interest or use-cases for this UDPExporter. So I'll try to give it a shot to see if the OTel Spec has a place for this component.

[jj22ee]

would this component be added to https://github.com/open-telemetry/opentelemetry-lambda?

To add more context, I want to keep the component contribution in this contrib repository, which is what this issue is for. The issue I created in the opentelemetry-lambda repo is to discuss adding the component as a configurable option in the Lambda Layers.

Question - Despite the given reason to have this "udp exporter" here... since it's specific for Lambda+XRay use case (at the moment), does that mean it shouldn't belong in this contrib repo? I wanted to start working on adding this component to the js and python contrib repos, but I want to know if it would be accepted.

[pichlermarc]

I was thinking it would make it easier for the OTel Lambda Layers to accept adding support for this UDPExporter component if it is hosted in an OTel repository.

Yes, that may be the case. Before adding it, it may be good to know if that's a requirement for the Lambda SIG though :)

I was also thinking of the possibility of making it available through the OTEL_EXPORTER_OTLP_PROTOCOL env var (but I guess this is unlikely to be accepted given this UDPExporter is not in OTel spec?).

Yes that is unlikely. OTEL_EXPORTER_OTLP_PROTOCOL is is a specified env var with certain well-known values. We would not add anything there as it stands today. OTEL_TRACES_EXPORTER may be an option, but from what I can tell both are not currently used by the OTel lambda layer - so us adding it here somewhere and instantiating it based on OTEL_TRACES_EXPORTER seems unlikely to fix your problem. 🤔

But speaking of the OTel Spec, I do wonder if anyone there has any interest or use-cases for this UDPExporter. So I'll try to give it a shot to see if the OTel Spec has a place for this component.

That sounds like a good idea. Seems like interest is rather low at the moment, though (ref https://github.com/open-telemetry/opentelemetry-specification/issues/4396) :/

To add more context, I want to keep the component contribution in this contrib repository, which is what this issue is for. The issue I created in the opentelemetry-lambda repo is to discuss adding the component as a configurable option in the Lambda Layers.

Question - Despite the given reason to have this "udp exporter" here... since it's specific for Lambda+XRay use case (at the moment), does that mean it shouldn't belong in this contrib repo? I wanted to start working on adding this component to the js and python contrib repos, but I want to know if it would be accepted.

You could definitely put it here as a vendor component if adhered to the requirements. But what I'm questioning is the following:

If the goal is not to add the exporter to any metapackage in this repo and if the Lambda SIG would not require you to have it in a OTel repo, then why put it here? For instance, there's vendor resource detectors in this repository. All of these are here because they want to be available as part of @opentelemetry/auto-instrumentations-node's getResourceDetectors(). If they would not want to be part of that, owning a seperate repository themselves would make all things a lot easier for the developers of the resource detectors because they can own the whole chain (reviews, release schedule, tooling) without having to deal with other package's problems delaying anything.

For you having it in a separate repo means you're able to iterate quicker on your components and get timely reviews from your colleagues. 🙂 For us means that any problems with it would not block us from releasing (with 60+ packages in the repo this is a real concern and happens regularly).