dd-trace-js

dd-trace-js copied to clipboard

Expected behaviour APM traces should contain logs

Actual behaviour Traces are gathered, but tracing logs section is empty

Steps to reproduce I tried with pino logger and also the simple log function from the example, examples:

import tracer from '../tracing/tracer';
import formats = require('dd-trace/ext/formats');

function log(level, message) {
  const span = tracer.scope().active();
  const time = new Date().toISOString();
  const record = { time, level, message };

  if (span) {
    tracer.inject(span.context(), formats.LOG, record);
  }

  console.log(JSON.stringify(record));
}

  @Get('test')
  testGet() {
    log('info', 'text log');
    log('error', 'test error');
    return 'test';
  }

import tracer from 'dd-trace';

// initialized in a different file to avoid hoisting.

tracer.init({
  logInjection: true,
  runtimeMetrics: true,
});

// enable and configure postgresql integration
tracer.use('pg', {
  service: 'pg-cluster',
});

// enable and configure postgresql integration
tracer.use('graphql', {
  service: 'graphql-server',
});

// enable and configure pino logger integration
tracer.use('pino', {
  service: 'pino-logger',
});

export default tracer;

Environment

• Operation system: Agent and app are hosted on ec2 amazon linux

• Node.js version: v16.16.0

• Tracer version: "dd-trace": "^2.12.2",

• Agent version: sudo datadog-agent status Agent (v7.38.2)

• Relevant library versions:

Tried with two pino versions, results in same effect:

• Latest 7x - pino": "^7.11.0
• Latest pino": "^8.4.1

Other deps

"nestjs-pino": "^3.1.1"
"@nestjs/core": "^8.0.0"
"@nestjs/graphql": "^10.0.16"
"@nestjs/apollo": "^10.0.16",
"graphql": "^16.5.0",

Agent configuration, it looks like it's receiving traces, but not logs:

==========
Logs Agent
==========
    Reliable: Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.eu on port 443
    BytesSent: 0
    EncodedBytesSent: 0
    LogsProcessed: 0
    LogsSent: 0

  container_collect_all
  ---------------------
    - Type: docker
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

=========
APM Agent
=========
  Status: Running
  Pid: 9186
  Uptime: 602 seconds
  Mem alloc: 14,342,744 bytes
  Hostname: i-062727d23d6b32491
  Receiver: 0.0.0.0:8126
  Endpoints:
    https://trace.agent.datadoghq.eu

  Receiver (previous minute)
  ==========================
    From nodejs v16.16.0 (v8), client 2.12.2
      Traces received: 85 (132,345 bytes)
      Spans received: 255

    Default priority sampling rate: 100.0%
    Priority sampling rate for 'service:backend-http-client,env:prod': 100.0%

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

How can I debug to solve the problem?

This library is able to inject the tracing context into your logs in order to correlate between traces and logs, but it doesn't actually send the logs on its own. This would need to be done either by sending the logs to stdout and configuring the agent to send these to Datadog, or by sending them directly with the HTTP transport.

Basically, logs should be sent the same way as if you didn't have dd-trace installed at all, and adding dd-trace simply means that in the UI traces will link to corresponding logs and vice versa.

You can find more information about submitting logs in the docs for Docker to forward stdout logs from containers, or in the Node documentation to submit logs directly from the app.

That clarifies a lot. Thank you!

One more question: I use AWS Elastic Beanstalk and all my logs (json - pino) are printed to stdout and finally goes to CloudWatch. Is there any integration which could transfer the logs from CloudWatch into DataDog? I would prefer not to overwhelm the node.js app server with submitting the logs via http. @rochdev

I'm no AWS expert, but I found the following 2 articles that seem to describe sending logs from AWS to Datadog and should apply to Beanstalk as well:

https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function/?tab=awsconsole https://aws.amazon.com/blogs/apn/how-to-automate-centralized-logging-and-integrate-with-datadog/

The best place to ask this is probably in the #logs channel of our public Slack since engineers from the Logs team will be able to answer directly.

Thank you, appreciate.

@rochdev Is it possible to send traces using dd-trace without agent, the same way as logs in Agentless logging?

@mariuszbeltowski You should be able to use either the Datadog Forwarder or the newer Datadog Extension depending on your configuration.

It looks like this issue was solved a while ago so I'll close it for now. Feel free to reopen if I'm wrong.