opentelemetry-lambda icon indicating copy to clipboard operation
opentelemetry-lambda copied to clipboard

Question: Collector not sending data

Open barendb opened this issue 4 years ago • 8 comments

Our organisation is running a few tests to see if this is a viable option, but we've run into some teething issues, the collector receives data, but none of that data is sent to exporters.

Just wondering what we can look at. We're using the following conf

receivers:
  otlp:
    protocols:
      grpc: # on port 55680
      http: # on port 55681

processors:
  batch:

exporters:
  otlp:
    endpoint: "https://api.honeycomb.io:443"
    headers:
      "x-honeycomb-team": "****redacted***"
      "x-honeycomb-dataset": "testing"

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

We can see the event is received, but never sent to our exporter

[opentelemetry-lambda-extension] Received event: {
	"eventType": "INVOKE",
	"deadlineMs": 1614115570609,
	"requestId": "dcaba70b-72e5-4b02-8bef-f7355fecd324",
	"invokedFunctionArn": "arn:aws:lambda:us-west-2:****redacted****:function:extention-test",
	"tracing": {
		"type": "X-Amzn-Trace-Id",
		"value": "Root=1-****redacted****;Sampled=0"
	}
}

barendb avatar Feb 23 '21 21:02 barendb

Tried with variant endpoint: "api.honeycomb.io:443", this gives us the same result

rickydua avatar Feb 23 '21 22:02 rickydua

Refer design doc, The Collector needs to remove all processors from config to keep one thread only from the receiver to the exporter. In this way, telemetry data is flushed to the backend synchronously at the end of every invocation.

wangzlei avatar May 03 '21 21:05 wangzlei

I'm having the same issue, without using processors:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
    loglevel: debug
  otlp:
    endpoint: ***some webhook***
    insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp]

Tried with otlp exporter and also with otlphttp exporter. @wangzlei can you please advice?

mirii1994 avatar Jun 21 '21 10:06 mirii1994

I'm experiencing the same issue where I can't export to honeycomb. I can do it locally just fine using the docker image otel/opentelemetry-collector.

longility avatar Aug 24 '21 22:08 longility

Any movement/advice on this? Seeing the same behavior.

brett-bim avatar Feb 02 '22 15:02 brett-bim

Any movement/advice on this? Seeing the same behavior.

Can you provide some additional detail the behavior you are seeing? Is it safe to assume you are seeing the logging exporter produced text output but do not see ingest into your otlp vendor target?

ericksoen avatar Feb 04 '22 20:02 ericksoen

This actually ended up being user error regarding where the AWS_LAMBDA_EXEC_WRAPPER was pointing. I'm using the AWS ADOT library and didn't have it configured properly. Sorry for the trouble.

brett-bim avatar Feb 07 '22 23:02 brett-bim