Help? Having trouble under Docker Compose getting trace information from my app into jaeger

[reselbob]

Hi:

I have a simple Node.js app the user the Open Telemetry Jaeger exporter to send trace information into Jaeger. It runs fine when I fire up the Jaeger as a Docker container then run the code from my machine's command line against localhost. However, when I try to run both the app and jaeger under Docker Compose within a Docker Compose network, the service registers and is apparent in the Jaeger UI, but the trace/span information never gets received into Jaeger.

Here is the code: https://github.com/reselbob/simpletracing/tree/releases/v1.0

I attached a screenshot to demonstrate that the service seems to be registering but the spans are not getting through. I am a bit at wit's end. Also, I hope I am complying with the contributing guidelines. My intentions are honest.

[dyladan]

Hmm nothing strikes me as obviously wrong with your docker-compose file. Have you tried enabling debug logging?

[reselbob]

@dyladan can you provide a link that tells me how to enable debug logging? Also, thanks for taking the time out of your day to help me out.

[dyladan]

Sure! There is no link I think, but the SDK exports a ConsoleLogger, which can be configured like this:

// ensure you require the logger
const { ConsoleLogger, LogLevel } = require("@opentelemetry/core");

// configure your desired log level
const logger = new ConsoleLogger(LogLevel.DEBUG);

// use the logger as part of your options when constructing SDK components
const provider = new NodeTracerProvider({
  logger: logger,
});

const options = {
  serviceName: serviceName,
  logger: logger
}
const exporter = new JaegerExporter(options);

[reselbob]

Thank you @dyladan !

Not seeing anything new with logger attached.

[reselbob]

@dyladan I am wondering if I could get some more detailed help from you. But, I don't feel at all comfortable asking for this help for free. If we can work out something, let's figure out a way to connect.

[DazWilkin]

I observe the same behavior using @reselbob's code and I'm unable to find issue with it (either).

I added @dyladan's ConsoleLogger but this generates no logs; I ran the jaegertracing/all-in-one container with --log-level=debug and this provided more details.

Using a single docker-compose file, I added HotRod (and a Golang example) and both correctly list when using "Find Traces".

I suspect that the Node.JS OpenTelemetry client is mostly (!) working, because I am able to enumerate traces reported by it, e.g.

http://localhost:16686/api/traces/f7d6ad69ba39d5bb61b84bbe12838121 yields:

NOTE The serviceName is RandomWordGenUpper

{
  "data": [{
    "traceID": "f7d6ad69ba39d5bb61b84bbe12838121",
    "spans": [{
      "traceID": "f7d6ad69ba39d5bb61b84bbe12838121",
      "spanID": "85969f3af93a5e65",
      "flags": 1,
      "operationName": "handleRequestToday",
      "references": [{
        "refType": "CHILD_OF",
        "traceID": "f7d6ad69ba39d5bb61b84bbe12838121",
        "spanID": "a874dc5e72995e86"
      }],
      "startTime": 197578396922,
      "duration": 291,
      "tags": [{
        "key": "time",
        "type": "string",
        "value": "Tue Oct 06 2020 20:30:25 GMT+0000 (UTC)"
      }, {
        "key": "status.code",
        "type": "float64",
        "value": 0
      }, {
        "key": "status.name",
        "type": "string",
        "value": "OK"
      }, {
        "key": "span.kind",
        "type": "string",
        "value": "SERVER"
      }, {
        "key": "telemetry.sdk.language",
        "type": "string",
        "value": "nodejs"
      }, {
        "key": "telemetry.sdk.name",
        "type": "string",
        "value": "opentelemetry"
      }, {
        "key": "telemetry.sdk.version",
        "type": "string",
        "value": "0.11.0"
      }, {
        "key": "internal.span.format",
        "type": "string",
        "value": "proto"
      }],
      "logs": [{
        "timestamp": 197578397000,
        "fields": [{
          "key": "message.id",
          "type": "string",
          "value": "invoking handleRequest"
        }]
      }, {
        "timestamp": 197578397000,
        "fields": [{
          "key": "0",
          "type": "string",
          "value": "H"
        }, {
          "key": "1",
          "type": "string",
          "value": "i"
        }, {
          "key": "2",
          "type": "string",
          "value": " "
        }, {
          "key": "3",
          "type": "string",
          "value": "T"
        }, {
          "key": "4",
          "type": "string",
          "value": "h"
        }, {
          "key": "5",
          "type": "string",
          "value": "e"
        }, {
          "key": "6",
          "type": "string",
          "value": "r"
        }, {
          "key": "7",
          "type": "string",
          "value": "e"
        }, {
          "key": "message.id",
          "type": "string",
          "value": "message"
        }]
      }],
      "processID": "p1",
      "warnings": null
    }, {
      "traceID": "f7d6ad69ba39d5bb61b84bbe12838121",
      "spanID": "a874dc5e72995e86",
      "flags": 1,
      "operationName": "GET /",
      "references": [],
      "startTime": 197578395634,
      "duration": 1010394,
      "tags": [{
        "key": "http.url",
        "type": "string",
        "value": "http://localhost:3000/"
      }, {
        "key": "http.host",
        "type": "string",
        "value": "localhost:3000"
      }, {
        "key": "net.host.name",
        "type": "string",
        "value": "localhost"
      }, {
        "key": "http.method",
        "type": "string",
        "value": "GET"
      }, {
        "key": "http.route",
        "type": "string",
        "value": "/"
      }, {
        "key": "http.target",
        "type": "string",
        "value": "/"
      }, {
        "key": "http.user_agent",
        "type": "string",
        "value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
      }, {
        "key": "http.flavor",
        "type": "string",
        "value": "1.1"
      }, {
        "key": "net.transport",
        "type": "string",
        "value": "IP.TCP"
      }, {
        "key": "net.host.ip",
        "type": "string",
        "value": "undefined"
      }, {
        "key": "net.host.port",
        "type": "string",
        "value": "undefined"
      }, {
        "key": "net.peer.ip",
        "type": "string",
        "value": "undefined"
      }, {
        "key": "net.peer.port",
        "type": "string",
        "value": "undefined"
      }, {
        "key": "http.status_code",
        "type": "float64",
        "value": 200
      }, {
        "key": "http.status_text",
        "type": "string",
        "value": "OK"
      }, {
        "key": "status.code",
        "type": "float64",
        "value": 0
      }, {
        "key": "status.name",
        "type": "string",
        "value": "OK"
      }, {
        "key": "span.kind",
        "type": "string",
        "value": "SERVER"
      }, {
        "key": "telemetry.sdk.language",
        "type": "string",
        "value": "nodejs"
      }, {
        "key": "telemetry.sdk.name",
        "type": "string",
        "value": "opentelemetry"
      }, {
        "key": "telemetry.sdk.version",
        "type": "string",
        "value": "0.11.0"
      }, {
        "key": "internal.span.format",
        "type": "string",
        "value": "proto"
      }],
      "logs": [],
      "processID": "p1",
      "warnings": null
    }],
    "processes": {
      "p1": {
        "serviceName": "RandomWordGenUpper",
        "tags": []
      }
    },
    "warnings": null
  }],
  "total": 0,
  "limit": 0,
  "offset": 0,
  "errors": null
}

While curl --silent http://localhost:16686/api/services includes the service:

{
  "data":[
    "RandomWordGenUpper",
    "jaeger-query"
  ],
  "total":2,
  "limit":0,
  "offset":0,
  "errors":null
}

This shows no traces curl --silent http://localhost:16686/api/traces?service=RandomWordGenUpper

{
  "data":[],
  "total":0,
  "limit":0,"
  offset":0,
  "errors":null
}

Even though curl --silent http://localhost:16686/api/traces?service=jaeger-query | jq '.data|length' returns 25 entries, i.e. these methods appear to reflect one another, listing services and then querying traces by service (ID).

So, I suspect the UI is also OK but it's unable to retrieve traces for the service RandomWordGenUpper when it tries to /api/traces?service=RandomWordGenUpper suggesting (!?) that the issue is that the Node.JS exporter is not perfectly generating traces. That these traces appear to correctly reflect the serviceName but somewhere there's an issue.

I noticed there's also an opentelemetry-all-in-one container image but this appears to function similarly.

docker-compose.yaml:

version: "3"
services:
  example:
    image: jaegertracing/example-hotrod:1.9
    container_name: hotrod
    depends_on:
      - jaeger
    environment:
      - JAEGER_AGENT_HOST=jaeger
    ports:
      - 8080-8083:8080-8083

  http:
    image: "64070408"
    container_name: http
    depends_on:
      - jaeger
    build:
      context: ./httpserver
      dockerfile: Dockerfile
    ports:
      - "3000:3000"
    environment:
      - JAEGER_AGENT_HOST=jaeger
      - JAEGER_AGENT_PORT=6832
      # - JAEGER_COLLECTOR_HOST=jaeger
      # - JAEGER_COLLECTOR_PORT=14268
      # - JAEGER_SAMPLER_TYPE=const
      # - JAEGER_SAMPLER_PARAM=1
      # - JAEGER_REPORTER_LOG_SPANS=true
      # - JAEGER_REPORTING_HOST=jaeger

  jaeger:
    image: jaegertracing/all-in-one:latest
    container_name: jaeger
    command:
      - --log-level=debug
    ports:
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "14268:14268"
      - "14269:14269"
      - "16686:16686"

[JavonneM]

Had a similar issue except none of the services showed up. Fixed by adding

extra_hosts:
      - "host.docker.internal:host-gateway"

in my apps service and specifying the jaeger endpoint with the following

const options = {
  tags: [],
  endpoint: http://host.docker.internal:14268/api/traces,
};
const exporter = new JaegerExporter(options);

This also worked for getting Prometheus targets to show that they are UP

Compose:

version: "3.6"
services:
  prometheus:
    image: prom/prometheus
    volumes:
      - type: bind
        source: ./prometheus.debug.yml
        target: /etc/prometheus/prometheus.yml
    ports:
      - 9090:9090
    extra_hosts:
      - "host.docker.internal:host-gateway"
  jaeger:
    image: jaegertracing/all-in-one:latest
    container_name: jaeger
    environment:
      - LOG_LEVEL=debug
      # - COLLECTOR_ZIPKIN_HTTP_PORT=9411
      # - LOG_LEVEL=debug
      # - JAEGER_SAMPLER_TYPE=const
      # - JAEGER_SAMPLER_PARAM=1
      # - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "5775:5775/udp"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778"
      - "16686:16686"
      - "14268:14268"
      - "9411:9411"
      - "14250:14250"
      - "14269:14269"
      - "14267:14267"
    extra_hosts:
      - "host.docker.internal:host-gateway"
  redis:
    image: redis
    container_name: cache
    expose:
      - 6379
  app:
    working_dir: /app
    image: node:14-alpine
    ports:
      - 3000:3000
      - 9229:9229
    command: npm run dev-container
    links:
      - jaeger
      - redis
    environment:
      - REDIS_HOST=redis
    volumes:
      - .:/app
    extra_hosts:
      - "host.docker.internal:host-gateway"