Problem with using logzio exporter in Lambda layer

[jkowall]

Hello, we have a user who is trying to use the Lambda layer with the logzio exporter that's been in the last releases. They were getting an error that the exporter doesn't exist. We noticed in the build manifest that it includes none of the 3rd party supported exporters which are not coming from AWS. I can submit a PR to fix this build if you'd like: https://github.com/aws-observability/aws-otel-collector/blob/main/pkg/lambdacomponents/defaults.go#L23-L26 Please advise.

[lupengamzn]

Hey @jkowall ,

Currently we don't have a plan to add the third party exporter to the ADOT collector for Lambda component, but we'll be working on that to enable adding other collector components from downstream. There is some workaround you can try at this moment:

1. Fork the aws-otel-collector repo and added your exporter in defaults.go and go.mod. Replace this line in go.mod with the intended module name and release it as a module similar to lambdacomponents.
2. Git clone this repo (git clone --recurse-submodules https://github.com/aws-observability/aws-otel-lambda.git) and replace the module in patch-upstream.sh with the module you released. Run ./patch-upstream.sh
3. Navigate to the SDK repo you're using and run ./build.sh
4. Navigate to the SDK’s sample app repo and build it using terraform. For instance, if you're using Python, then:

cd integration-tests/python/aws-sdk
terraform init
terraform apply -auto-approve

You should be able to see the Lambda layer ARN logged out in terminal and apply that to your Lambda application to enable logzio exporter.

[jkowall]

Thanks for the response @lupengamzn

What's the purpose of getting added to the AWS collector when you are going to remove the exporters from the distribution at will. That seems like a pointless exercise we went through to be added to this distribution. I am very disappointed in this process and the way AWS manages the open-source community. It's not the way it should be working.

[wangzlei]

Sorry to cause trouble. There are 2 reasons that we cannot add all of exporters in AWS managed OpenTelemetry layer.

The main reason is a technical trouble issued in Design 2.2 and 2.3. A full Collector would exhaust 150MB+ memory in user's Lambda sandbox, but one user only need 1 or 2 exporters. so, we create a lambdacomponents module to strip down the Collector in Lambda layer. That means there is no one OpenTelemetry layer would satisfy all of users, user has to custom the layer based on his demand by replacing lambdacomponents. That is how AWS managed lambda layer come as a downstream Repo.

The second reason is AWS managed Lambda layer is not under OpenTelemetry community, any component to be added in must have passed security check, for example, there is end-2-end verification for xray exporter and daily canary test to make sure the released layer is available in every AWS region. Ideally, as a downstream repo of opentelemetry-lambda, AWS managed layer wants to support 3rd party exporter as more as possible. Honestly we hope power contributor can help us add more 3rd party exporter. The basic principles are: 1) not inflate the size of layer too much; 2) add e2e test for that exporter in CI/CD if possible.

So, AWS managed Lambda layer is just a downstream of opentelemetry-lambda, not under opentelemetry community directly. We will add more 3rd party exporters in near future if that passes the security review. Before that an alternative solution is user builds his custom layer on demand. I also cut an issue in opentelemetry-lambda that ask for providing an easy way to build custom layer. https://github.com/open-telemetry/opentelemetry-lambda/issues/100

[jkowall]

Thanks for all of the detail here. The process to use Teraform must be done by each of our customers for each Lambda account. This is not really feasible. We're going to just use EC2 which eliminates the point of the layers. It's really unfortunate for users who aren't using x-ray. I understand the constraints, but the design should have been pluggable in some easier way versus just including the x-ray exporter. This is showing a bias towards the AWS tracing solution and not with any partner solution.

When we added the exporter to the AWS otel distro we had to jump through a lot of hoops for reviews both security, support, testing, and much more. We kind of expected that work to be included in the AWS services, and when the customer was unable to use it in a lambda layer it was pretty disappointing. I can't figure out why we bothered to do that work if we don't get the advantages within AWS.

To me this highlights yet another reason that AWS doesn't get the way open source should be working, it's about a community of technologies and not the AWS solution to the problem. I'll bring this up with our sponsors as this is by no means the first time we've seen this issue.

[anuraaga]

@jkowall Thanks for bringing this up - I agree this situation is far from ideal. Really, if the customer's function memory limit was not eaten up by the larger collector size we definitely would just go with the full aws-otel-collector distro. On the bright side, I don't see why we wouldn't do that for any future managed solution based on ADOT that may pop up.

I was wondering what would be a good solution for logzio customers? Would publishing another version of each layer with the full aws-otel-collector, with the caveat of the memory usage, be ok? Given that it's sort of a reasonable expectation for customers that aws-otel-lambda would have the featureset of aws-otel-collector this seems like a good thing to do either way in hindsight.

[jkowall]

That's good to hear, but it doesn't really give us a good solution right now. The current plan is to suggest they do not use the layer and instead use an EC2 collector to forward the data. It's not ideal, but it is what it is.

I'm not a Lambda expert as yourself, so I am not sure what the impact is for the larger memory usage or how that will affect the general functioning of the Lambda functions. Is there higher hosts, slower performance?

[wangzlei]

Refer https://github.com/aws-observability/aws-otel-lambda/issues/104

[wangzlei]

That's good to hear, but it doesn't really give us a good solution right now. The current plan is to suggest they do not use the layer and instead use an EC2 collector to forward the data. It's not ideal, but it is what it is.

I'm not a Lambda expert as yourself, so I am not sure what the impact is for the larger memory usage or how that will affect the general functioning of the Lambda functions. Is there higher hosts, slower performance?

AWS Lambda charge user by Lambda sandbox capacity and execution time. If a Lambda layer eat too many resources, customer pay for that. When the memory consumption hit the red line, user see longer cold start than before, even get failed to launch unless upgrading to higher capacity.

One proposal in OTel Lambda SIG meeting is opentelemetry-lambda just contain collector core components in opentelemetry-collector. 3rd party companies not following OTLP, such as AWS, maintain their downstream repos/lambda layers respectively for their customers by customize the Collector extension. At the same time, opentelemetry-lambda provides an easy operated tool to build custom Lambda layer based on demand.

opentelemetry-lambda (publish layer with core components such as otlp exporters)
 |__aws_obersvability(publish layer with additional aws exporters)
 |_ xxx (publish layer with xxx exporters)
 |_??? (publish layer with logzip exporter ?)

So based on this model, any one can maintain a downstream lambda layer distro, either by user himself or by a company for its customers.

The reality of this model is that Lambda is AWS service, AWS Lambda Distro naturally would get more customer expectations, is wrongly thought AWS managed Lambda layer should support all components in opentelemetry. But that is a misunderstanding :)

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled