opentelemetry-js
opentelemetry-js copied to clipboard
Help? Having trouble under Docker Compose getting trace information from my app into jaeger
Hi:
I have a simple Node.js app the user the Open Telemetry Jaeger exporter to send trace information into Jaeger. It runs fine when I fire up the Jaeger as a Docker container then run the code from my machine's command line against localhost
. However, when I try to run both the app and jaeger under Docker Compose within a Docker Compose network, the service registers and is apparent in the Jaeger UI, but the trace/span information never gets received into Jaeger.
Here is the code: https://github.com/reselbob/simpletracing/tree/releases/v1.0
I attached a screenshot to demonstrate that the service seems to be registering but the spans are not getting through. I am a bit at wit's end. Also, I hope I am complying with the contributing guidelines. My intentions are honest.
Hmm nothing strikes me as obviously wrong with your docker-compose file. Have you tried enabling debug logging?
@dyladan can you provide a link that tells me how to enable debug logging? Also, thanks for taking the time out of your day to help me out.
Sure! There is no link I think, but the SDK exports a ConsoleLogger, which can be configured like this:
// ensure you require the logger
const { ConsoleLogger, LogLevel } = require("@opentelemetry/core");
// configure your desired log level
const logger = new ConsoleLogger(LogLevel.DEBUG);
// use the logger as part of your options when constructing SDK components
const provider = new NodeTracerProvider({
logger: logger,
});
const options = {
serviceName: serviceName,
logger: logger
}
const exporter = new JaegerExporter(options);
Thank you @dyladan !
Not seeing anything new with logger attached.
@dyladan I am wondering if I could get some more detailed help from you. But, I don't feel at all comfortable asking for this help for free. If we can work out something, let's figure out a way to connect.
I observe the same behavior using @reselbob's code and I'm unable to find issue with it (either).
I added @dyladan's ConsoleLogger
but this generates no logs; I ran the jaegertracing/all-in-one
container with --log-level=debug
and this provided more details.
Using a single docker-compose file, I added HotRod (and a Golang example) and both correctly list when using "Find Traces".
I suspect that the Node.JS OpenTelemetry client is mostly (!) working, because I am able to enumerate traces reported by it, e.g.
http://localhost:16686/api/traces/f7d6ad69ba39d5bb61b84bbe12838121
yields:
NOTE The
serviceName
isRandomWordGenUpper
{
"data": [{
"traceID": "f7d6ad69ba39d5bb61b84bbe12838121",
"spans": [{
"traceID": "f7d6ad69ba39d5bb61b84bbe12838121",
"spanID": "85969f3af93a5e65",
"flags": 1,
"operationName": "handleRequestToday",
"references": [{
"refType": "CHILD_OF",
"traceID": "f7d6ad69ba39d5bb61b84bbe12838121",
"spanID": "a874dc5e72995e86"
}],
"startTime": 197578396922,
"duration": 291,
"tags": [{
"key": "time",
"type": "string",
"value": "Tue Oct 06 2020 20:30:25 GMT+0000 (UTC)"
}, {
"key": "status.code",
"type": "float64",
"value": 0
}, {
"key": "status.name",
"type": "string",
"value": "OK"
}, {
"key": "span.kind",
"type": "string",
"value": "SERVER"
}, {
"key": "telemetry.sdk.language",
"type": "string",
"value": "nodejs"
}, {
"key": "telemetry.sdk.name",
"type": "string",
"value": "opentelemetry"
}, {
"key": "telemetry.sdk.version",
"type": "string",
"value": "0.11.0"
}, {
"key": "internal.span.format",
"type": "string",
"value": "proto"
}],
"logs": [{
"timestamp": 197578397000,
"fields": [{
"key": "message.id",
"type": "string",
"value": "invoking handleRequest"
}]
}, {
"timestamp": 197578397000,
"fields": [{
"key": "0",
"type": "string",
"value": "H"
}, {
"key": "1",
"type": "string",
"value": "i"
}, {
"key": "2",
"type": "string",
"value": " "
}, {
"key": "3",
"type": "string",
"value": "T"
}, {
"key": "4",
"type": "string",
"value": "h"
}, {
"key": "5",
"type": "string",
"value": "e"
}, {
"key": "6",
"type": "string",
"value": "r"
}, {
"key": "7",
"type": "string",
"value": "e"
}, {
"key": "message.id",
"type": "string",
"value": "message"
}]
}],
"processID": "p1",
"warnings": null
}, {
"traceID": "f7d6ad69ba39d5bb61b84bbe12838121",
"spanID": "a874dc5e72995e86",
"flags": 1,
"operationName": "GET /",
"references": [],
"startTime": 197578395634,
"duration": 1010394,
"tags": [{
"key": "http.url",
"type": "string",
"value": "http://localhost:3000/"
}, {
"key": "http.host",
"type": "string",
"value": "localhost:3000"
}, {
"key": "net.host.name",
"type": "string",
"value": "localhost"
}, {
"key": "http.method",
"type": "string",
"value": "GET"
}, {
"key": "http.route",
"type": "string",
"value": "/"
}, {
"key": "http.target",
"type": "string",
"value": "/"
}, {
"key": "http.user_agent",
"type": "string",
"value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
}, {
"key": "http.flavor",
"type": "string",
"value": "1.1"
}, {
"key": "net.transport",
"type": "string",
"value": "IP.TCP"
}, {
"key": "net.host.ip",
"type": "string",
"value": "undefined"
}, {
"key": "net.host.port",
"type": "string",
"value": "undefined"
}, {
"key": "net.peer.ip",
"type": "string",
"value": "undefined"
}, {
"key": "net.peer.port",
"type": "string",
"value": "undefined"
}, {
"key": "http.status_code",
"type": "float64",
"value": 200
}, {
"key": "http.status_text",
"type": "string",
"value": "OK"
}, {
"key": "status.code",
"type": "float64",
"value": 0
}, {
"key": "status.name",
"type": "string",
"value": "OK"
}, {
"key": "span.kind",
"type": "string",
"value": "SERVER"
}, {
"key": "telemetry.sdk.language",
"type": "string",
"value": "nodejs"
}, {
"key": "telemetry.sdk.name",
"type": "string",
"value": "opentelemetry"
}, {
"key": "telemetry.sdk.version",
"type": "string",
"value": "0.11.0"
}, {
"key": "internal.span.format",
"type": "string",
"value": "proto"
}],
"logs": [],
"processID": "p1",
"warnings": null
}],
"processes": {
"p1": {
"serviceName": "RandomWordGenUpper",
"tags": []
}
},
"warnings": null
}],
"total": 0,
"limit": 0,
"offset": 0,
"errors": null
}
While curl --silent http://localhost:16686/api/services
includes the service:
{
"data":[
"RandomWordGenUpper",
"jaeger-query"
],
"total":2,
"limit":0,
"offset":0,
"errors":null
}
This shows no traces curl --silent http://localhost:16686/api/traces?service=RandomWordGenUpper
{
"data":[],
"total":0,
"limit":0,"
offset":0,
"errors":null
}
Even though curl --silent http://localhost:16686/api/traces?service=jaeger-query | jq '.data|length'
returns 25 entries, i.e. these methods appear to reflect one another, listing services and then querying traces by service (ID).
So, I suspect the UI is also OK but it's unable to retrieve traces for the service RandomWordGenUpper
when it tries to /api/traces?service=RandomWordGenUpper
suggesting (!?) that the issue is that the Node.JS exporter is not perfectly generating traces. That these traces appear to correctly reflect the serviceName
but somewhere there's an issue.
I noticed there's also an opentelemetry-all-in-one
container image but this appears to function similarly.
docker-compose.yaml
:
version: "3"
services:
example:
image: jaegertracing/example-hotrod:1.9
container_name: hotrod
depends_on:
- jaeger
environment:
- JAEGER_AGENT_HOST=jaeger
ports:
- 8080-8083:8080-8083
http:
image: "64070408"
container_name: http
depends_on:
- jaeger
build:
context: ./httpserver
dockerfile: Dockerfile
ports:
- "3000:3000"
environment:
- JAEGER_AGENT_HOST=jaeger
- JAEGER_AGENT_PORT=6832
# - JAEGER_COLLECTOR_HOST=jaeger
# - JAEGER_COLLECTOR_PORT=14268
# - JAEGER_SAMPLER_TYPE=const
# - JAEGER_SAMPLER_PARAM=1
# - JAEGER_REPORTER_LOG_SPANS=true
# - JAEGER_REPORTING_HOST=jaeger
jaeger:
image: jaegertracing/all-in-one:latest
container_name: jaeger
command:
- --log-level=debug
ports:
- "6831:6831/udp"
- "6832:6832/udp"
- "14268:14268"
- "14269:14269"
- "16686:16686"
Had a similar issue except none of the services showed up. Fixed by adding
extra_hosts:
- "host.docker.internal:host-gateway"
in my apps service and specifying the jaeger endpoint with the following
const options = {
tags: [],
endpoint: http://host.docker.internal:14268/api/traces,
};
const exporter = new JaegerExporter(options);
This also worked for getting Prometheus targets to show that they are UP
Compose:
version: "3.6"
services:
prometheus:
image: prom/prometheus
volumes:
- type: bind
source: ./prometheus.debug.yml
target: /etc/prometheus/prometheus.yml
ports:
- 9090:9090
extra_hosts:
- "host.docker.internal:host-gateway"
jaeger:
image: jaegertracing/all-in-one:latest
container_name: jaeger
environment:
- LOG_LEVEL=debug
# - COLLECTOR_ZIPKIN_HTTP_PORT=9411
# - LOG_LEVEL=debug
# - JAEGER_SAMPLER_TYPE=const
# - JAEGER_SAMPLER_PARAM=1
# - COLLECTOR_OTLP_ENABLED=true
ports:
- "5775:5775/udp"
- "6831:6831/udp"
- "6832:6832/udp"
- "5778:5778"
- "16686:16686"
- "14268:14268"
- "9411:9411"
- "14250:14250"
- "14269:14269"
- "14267:14267"
extra_hosts:
- "host.docker.internal:host-gateway"
redis:
image: redis
container_name: cache
expose:
- 6379
app:
working_dir: /app
image: node:14-alpine
ports:
- 3000:3000
- 9229:9229
command: npm run dev-container
links:
- jaeger
- redis
environment:
- REDIS_HOST=redis
volumes:
- .:/app
extra_hosts:
- "host.docker.internal:host-gateway"