aws-otel-lambda icon indicating copy to clipboard operation
aws-otel-lambda copied to clipboard

Failure to export from collector to backend (GO Lambda Layer)

Open itaybaror opened this issue 3 years ago • 8 comments

I have been attempting to instrument lambdas using the docs on the ADOT website. I have successfully instrumented some python lambdas, where they send the data to the collector, and then exports the data to lightstep (my backend). However when i try to instrument in go, i do see the logs in cloudwatch (i configured the collector to send data to lightstep and to logs) but when it tries to export the data to lightstep (and i have tried other backends) i am usually greeted with some sort of grpc or timeout error, and i do not see the data in the backend.

I have attached pictures with the errors i receive. Keep in mind that with my working python implementation i am using the same collector config file.

Errors: Screen Shot 2022-07-25 at 10 04 16 AM

Screen Shot 2022-07-25 at 10 12 26 AM

My collector config file: Screen Shot 2022-07-25 at 10 11 41 AM

itaybaror avatar Jul 25 '22 14:07 itaybaror

You are using wrong exporter. Lightstep doesn't support OTLP GRPC, only OTLP HTTP - https://docs.lightstep.com/docs/send-otlp-over-http-to-lightstep

Correct exporter config should be:

  otlphttp:
    traces_endpoint: https://ingest.lightstep.com/traces/otlp/v0.9
    headers:
        'Lightstep-Access-Token': 'YOUR-LIGHTSTEP-ACCESS-TOKEN-HERE'

jangaraj avatar Jul 26 '22 17:07 jangaraj

@jangaraj Are you sure? i have seen in multiple places that oltp grpc should work. For example on the ADOT documentation... https://aws-otel.github.io/docs/components/otlp-exporter#lightstep

Screen Shot 2022-07-26 at 2 14 02 PM

itaybaror avatar Jul 26 '22 18:07 itaybaror

I'm sure. I would believe vendor doc in this case. Ask your Lightstep support.

jangaraj avatar Jul 26 '22 18:07 jangaraj

@jangaraj But otlp works to export to lighstep using my python implementation, and i dont think it explicitly says that you cant use it on the link you sent me.

itaybaror avatar Jul 26 '22 18:07 itaybaror

Also i get the about the same error when i try to configure olthttp Screen Shot 2022-07-26 at 2 32 47 PM

itaybaror avatar Jul 26 '22 18:07 itaybaror

I guess you don't have network connectivity (sec. group, network acl, NAT,...) to public LS collector from your Lambda.

jangaraj avatar Jul 26 '22 20:07 jangaraj

FYI, OTLP GRPC works to Lightstep. ingest.lightstep.com:443 in the Collector config works fine. It's strange that it would work from your Python Lambda but not your Go one, the only thing I can think would be on the AWS side, but if your two Lambdas are in the same security group and have the same ACLs and NAT and whatnot, then I'm kind of at a loss.

austinlparker avatar Aug 04 '22 02:08 austinlparker

Are there any updates to this issue? I'm experiencing something similar where my lambda functions work fine with otel lambda layers (Java, Node.js, Python) but fails to export traces with the Go lambda layer

stephenhong avatar Sep 07 '22 16:09 stephenhong

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled

github-actions[bot] avatar Dec 11 '22 20:12 github-actions[bot]

This issue was closed because it has been marked as stale for 30 days with no activity.

github-actions[bot] avatar Jan 15 '23 20:01 github-actions[bot]