Not able to export traces to OTLP exporter via AWS managed Lambda layer

[NithyaAhila]

We are trying to auto instrument our java lambda (using micronaut framework) using the lambda layer "arn:aws:lambda:<region>:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-10-1:1" as specified in https://aws-otel.github.io/docs/getting-started/lambda/lambda-java.

We have configured the following environment variables for the OTLP exporter, AWS_LAMBDA_EXEC_WRAPPER | /opt/otel-handler OTEL_EXPORTER_OTLP_ENDPOINT |https::20443 OTEL_EXPORTER_OTLP_HEADERS | Authorization=Bearer OTEL_EXPORTER_OTLP_TIMEOUT | 30000 OTEL_INSTRUMENTATION_AWS_LAMBDA_FLUSH_TIMEOUT | 30000 OTEL_INSTRUMENTATION_AWS_LAMBDA_HANDLER | io.micronaut.function.aws.MicronautRequestHandler::handleRequest OTEL_LOG_LEVEL | debug OTEL_METRICS_EXPORTER | otlp OTEL_PROPAGATORS | tracecontext,b3,b3multi,ottrace,xray,jaeger OTEL_RESOURCE_ATTRIBUTES | service.name=service,service.namespace=namespace,service.version=LATEST OTEL_SERVICE_NAME | serviceName OTEL_TRACES_EXPORTER | otlp OTEL_TRACES_SAMPLER | always_on

The traces are pushed to the AWS x-ray when we enabled the active tracing but its not pushing to the OTLP endpoint. From the cloudwatch logs, we could see there are only two exporters displayed - logging and awsxray but no exporter for OLTP.

Sample logs: builder/exporters_builder.go:255 Exporter was built. { "kind": "exporter", "name": "logging" }

builder/exporters_builder.go:255 Exporter was built. { "kind": "exporter", "name": "awsxray" }

Do we need additional configurations or a separate collector and exporter for the OTLP? As per the documentation the lambda layer has the java agent as well the collector inbuilt for OTLP from https://github.com/aws-observability/aws-otel-collector/blob/main/pkg/lambdacomponents/defaults.go#L23-L26

[NathanielRN]

Hi @NithyaAhila! Thanks for posting this issue and sorry for the delay.

I think the issue for why you may not be seeing logs indicating that an oltp exporter has started is because you have to provide your own custom configuration file. This config files is for the ADOT Collector that is included in the Java Lambda Layer. In the same guide you posted, there is a section about Configuring the ADOT Collector in the Lambda Layer with different components.

If you include a custom configuration, you'll end up with the default ADOT Collector configuration which only includes the logging and awsxray exporters.

Let me know if that makes sense!

Also just want to clarify this comment:

We have configured the following environment variables for the OTLP exporter, ... OTEL_EXPORTER_OTLP_ENDPOINT |https::20443 OTEL_EXPORTER_OTLP_HEADERS | Authorization=Bearer OTEL_EXPORTER_OTLP_TIMEOUT | 30000 ... OTEL_TRACES_EXPORTER | otlp

You cannot configure the OTLP Exporter on the Collector like this. This is for the OTLP exporter on the Java Lambda App you are trying to instrument.

You have to use the configuration file I mentioned above for this. Check out this OTel Collector documentation to learn how to configure the otlp Collector Exporter.

[NithyaAhila]

Thanks for the inputs @NathanielRN . Will check with the custom configurations.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled

[NithyaAhila]

We tried with the environment variables. We could see the startup logs getting pushed but we are not able to send the custom span/traces.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled

[github-actions[bot]]

This issue was closed because it has been marked as stale for 30 days with no activity.