opentelemetry-collector-contrib
opentelemetry-collector-contrib copied to clipboard
open-telemetry
OTEL Collector not able to connect to Elastic APM | Docker
Describe the bug OTEL Collector not able to connect to elastic-apm
Steps to reproduce
Please follow the instructions here, it is simple App https://github.com/sathishsoundharajan/otel-elastic-apm
What did you expect to see? i'm expecting to see the tracing logs to be sent both to zipkin and elastic-apm server.
What did you see instead?
otel-collector | 2022-07-29T14:39:44.207Z error exporterhelper/queued_retry.go:149 Exporting failed. Try enabling retry_on_failure config option to retry on retryable errors {"kind": "exporter", "data_type": "traces", "name": "elastic", "error": "sending event request failed: Post \"http://elastic-apm:8200/intake/v2/events\": dial tcp 192.168.32.6:8200: connect: connection refused", "errorVerbose": "Post \"http://elastic-apm:8200/intake/v2/events\": dial tcp 192.168.32.6:8200: connect: connection refused\nsending event request failed\ngo.elastic.co/apm/transport.(*HTTPTransport).sendStreamRequest\n\tgo.elastic.co/[email protected]/transport/http.go:292\ngo.elastic.co/apm/transport.(*HTTPTransport).SendStream\n\tgo.elastic.co/[email protected]/transport/http.go:282\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter.(*elasticExporter).sendEvents\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/exporter.go:197\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter.(*elasticExporter).ExportResourceSpans\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/exporter.go:149\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter.newElasticTracesExporter.func1\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/exporter.go:53\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).export\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/traces.go:70\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:225\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:147\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/traces.go:134\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:83\ngo.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2\n\tgo.opentelemetry.io/[email protected]/exporter/exporterhelper/traces.go:113\ngo.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces\n\tgo.opentelemetry.io/[email protected]/consumer/traces.go:36\ngo.opentelemetry.io/collector/service/internal/fanoutconsumer.(*tracesConsumer).ConsumeTraces\n\tgo.opentelemetry.io/[email protected]/service/internal/fanoutconsumer/traces.go:75\ngo.opentelemetry.io/collector/processor/batchprocessor.(*batchTraces).export\n\tgo.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:262\ngo.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).sendItems\n\tgo.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:176\ngo.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle\n\tgo.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:143\nruntime.goexit\n\truntime/asm_amd64.s:1571"}
otel-collector | go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
otel-collector | go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:149
otel-collector | go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
otel-collector | go.opentelemetry.io/[email protected]/exporter/exporterhelper/traces.go:134
otel-collector | go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send
otel-collector | go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:83
otel-collector | go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2
otel-collector | go.opentelemetry.io/[email protected]/exporter/exporterhelper/traces.go:113
otel-collector | go.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces
otel-collector | go.opentelemetry.io/[email protected]/consumer/traces.go:36
otel-collector | go.opentelemetry.io/collector/service/internal/fanoutconsumer.(*tracesConsumer).ConsumeTraces
otel-collector | go.opentelemetry.io/[email protected]/service/internal/fanoutconsumer/traces.go:75
otel-collector | go.opentelemetry.io/collector/processor/batchprocessor.(*batchTraces).export
otel-collector | go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:262
otel-collector | go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).sendItems
otel-collector | go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:176
otel-collector | go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle
otel-collector | go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:143
What version did you use? otel/otel-collector-contrib:Latest docker-image.. Please check the docker-compose.yml file mentioned in the above repo.
What config did you use? Config: (e.g. the yaml config file)
Environment Docker
Additional context What I'm trying to do is setup small node.js application
- Collect the traces via OTEL Collector
- Make sure OTEL Collector sends data to Zipkin and as well as Elastic APM
Yes @codeboten i logged into the container using docker exec -it <container_id> sh and executed curl http://localhost:8200/ and got response "Ok" and build_number. But not able to somehow access it outside the container and also OTEL is not able to connect. Am i missing something very obvious.
I have added more information in these threads: https://discuss.elastic.co/t/elastic-apm-not-accessible-outside-docker-container/311026 https://stackoverflow.com/questions/73173816/otel-collector-not-able-to-connect-to-elastic-apm-docker
Fixing the host parameter in apm-server.yml to "0.0.0.0:8200" worked.
But as mentioned in other threads, i have tried to use the latest elastic exporter
receivers:
zipkin:
otlp:
protocols:
grpc:
http:
processors:
batch:
timeout: 10s
exporters:
logging:
logLevel: debug
zipkin:
endpoint: "http://zipkin:9411/api/v2/spans"
otlp/elastic:
endpoint: http://elastic-apm:8200
tls:
insecure: true
extensions:
health_check:
service:
extensions: [health_check]
pipelines:
# metrics:
# receivers: [ otlp ]
traces:
receivers: [ zipkin, otlp ]
processors: [ batch ]
exporters: [ zipkin, otlp/elastic ]
docker-compose.yml
version: '3'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.23
container_name: elasticsearch
environment:
- xpack.security.enabled=false
- discovery.type=single-node
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
cap_add:
- IPC_LOCK
volumes:
- elasticsearch-data:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
healthcheck:
test: ["CMD-SHELL", "curl -s http://localhost:9200/_cluster/health | grep -vq '\"status\":\"red\"'"]
retries: 10
interval: 20s
networks:
- opentelemetry
kibana:
container_name: kibana
image: docker.elastic.co/kibana/kibana:6.8.23
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
ports:
- 5601:5601
depends_on:
elasticsearch:
condition: service_healthy
healthcheck:
test: ["CMD", "curl", "--write-out", "'HTTP %{http_code}'", "--silent", "--output", "/dev/null", "http://localhost:5601/"]
retries: 10
interval: 10s
networks:
- opentelemetry
elastic-apm:
container_name: elastic-apm
build:
context: ../
dockerfile: ./build/elastic-apm/Dockerfile
ports:
- 8200:8200
cap_drop:
- ALL
cap_add:
- CHOWN
- DAC_OVERRIDE
- SETGID
- SETUID
depends_on:
elasticsearch:
condition: service_healthy
kibana:
condition: service_healthy
healthcheck:
interval: 10s
retries: 12
test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/
networks:
- opentelemetry
zipkin:
image: openzipkin/zipkin
container_name: zipkin
ports:
- 9411:9411
networks:
- opentelemetry
otel-collector:
container_name: otel-collector
build:
context: ../
dockerfile: ./build/collector/Dockerfile
ports:
- 4318:4318
- 13133:13133
networks:
- opentelemetry
volumes:
elasticsearch-data:
driver: local
networks:
opentelemetry:
driver: bridge
When i start the containers i getting
otel-collector | 2022-08-01T11:22:09.861Z info pipelines/pipelines.go:86 Starting processors...
otel-collector | 2022-08-01T11:22:09.862Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "batch", "pipeline": "traces"}
otel-collector | 2022-08-01T11:22:09.862Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "batch", "pipeline": "traces"}
otel-collector | 2022-08-01T11:22:09.862Z info pipelines/pipelines.go:98 Starting receivers...
otel-collector | 2022-08-01T11:22:09.862Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "zipkin", "pipeline": "traces"}
otel-collector | 2022-08-01T11:22:09.863Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "zipkin", "pipeline": "traces"}
otel-collector | 2022-08-01T11:22:09.863Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
elasticsearch | [2022-08-01T11:22:25,317][INFO ][o.e.p.PluginsService ] [QyYZpCM] loaded module [analysis-common]
elasticsearch | [2022-08-01T11:22:25,318][INFO ][o.e.p.PluginsService ] [QyYZpCM] loaded module [ingest-common]
otel-collector | 2022-08-01T11:22:09.865Z info otlpreceiver/otlp.go:70 Starting GRPC server on endpoint 0.0.0.0:4317 {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
otel-collector | 2022-08-01T11:22:09.866Z info otlpreceiver/otlp.go:88 Starting HTTP server on endpoint 0.0.0.0:4318 {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
otel-collector | 2022-08-01T11:22:09.866Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
otel-collector | 2022-08-01T11:22:09.866Z info healthcheck/handler.go:129 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"}
otel-collector | 2022-08-01T11:22:09.866Z info service/collector.go:215 Starting otelcol-contrib... {"Version": "0.56.0", "NumCPU": 4}
otel-collector | 2022-08-01T11:22:09.866Z info service/collector.go:128 Everything is ready. Begin running and processing data.
elasticsearch | [2022-08-01T11:22:25,318][INFO ][o.e.p.PluginsService ] [QyYZpCM] loaded module [ingest-geoip]
otel-collector | 2022-08-01T11:22:09.896Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
otel-collector | "Addr": "elastic-apm:8200",
otel-collector | "ServerName": "elastic-apm:8200",
otel-collector | "Attributes": null,
otel-collector | "BalancerAttributes": null,
otel-collector | "Type": 0,
otel-collector | "Metadata": null
otel-collector | }. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup elastic-apm on 127.0.0.11:53: no such host" {"grpc_log": true}
Have updated the repro repo with latest configuration to reproduce
Pinging code owners: @axw @simitt @jalvz. See Adding Labels via Comments if you do not have permissions to add labels yourself.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
We have been discussing this at https://discuss.elastic.co/t/elastic-apm-not-accessible-outside-docker-container/311026, and I believe this is a configuration issue. I suggest we close this, continue the discussion on the linked forum post if necessary, and reopen if there turns out to be a bug.
