MongoDb traces no longer showing after mongodb driver upgrade

[sattarab]

We recently updated our mongodb driver from 3.6.4 to 4.3.1. Now the MongoDb traces are no longer being shown in the Datadog dashboard

Expected behaviour Updating the MongoDb library to 4.3.1 the traces should still work

Actual behaviour Updating the MongoDb library to 4.3.1 the traces have stopped working for MongoDb

Steps to reproduce Set the mongodb library version to 4.3.1 in package.json

"mongodb": "4.3.1",

Environment

• Operation system: Amazon Linux
• Node.js version: 14.7.3
• Tracer version: 2.5
• Agent version: 6
• Relevant library versions: "mongodb": "4.3.1"

[rochdev]

Looking at the nightly test it looks like our tests are passing for 4.5.0. Can you share a reproduction snippet?

[sattarab]

@rochdev we are using the following way in our code in coffeescript

tracer = require( 'dd-trace' ).init {
    analytics: true
    logInjection: true
    runtimeMetrics: true
    service: "web"
    trackAsyncScope: false
    tags:
      app: "web"
      env: "staging"
  }

tracer.use 'http', {
 blacklist: ignored_routes
}

tracer.use 'express', {
  blacklist: ignored_routes
}

Note that some of the options are outdated here as we were using version 0.31 of the library prior when mongodb traces was working for us

[rochdev]

Can you share runnable code that reproduces the issue? On my end everything works as it should.

[sattarab]

@rochdev You can use the following code to reproduce app.js

const { tracer } = require('dd-trace');

tracer.init({
  env: 'dev-sattarab',
  service: 'mongodb-dev',
  version: 'dev-sattarab'
});

const { MongoClient } = require('mongodb');

const url = 'mongodb://localhost:27017';
const client = new MongoClient(url);
const dbName = 'myProject';

async function main() {
  // Use connect method to connect to the server
  await client.connect();
  console.log('Connected successfully to server');
  const db = client.db(dbName);
  const collection = db.collection('documents');
  await collection.insertOne( { name: "test" } )
  const count = await collection.countDocuments()
  const documents = await collection.find().toArray()
  console.log( "documents", documents )
  console.log('count', count)

  return 'done.';
}

main()
  .then(console.log)
  .catch(console.error)
  .finally(() => client.close());

package.json

{
  "name": "mongo-tracer",
  "dependencies": {
    "dd-trace": "2.5.0",
    "mongodb": "4.5.0"
  }
}

Can run like node app.js

Note I tried it with Datadog agent 7 as well

Screenshot

[sattarab]

@rochdev any update on this?

[rochdev]

I've just tried the above code locally and I'm getting traces. Could it be related to the CoffeeScript transpilation? Did you try the above pure JavaScript version as well?

[sattarab]

@rochdev I did try the above javascript code that I shared with you. The above screenshot is what I got. I am not sure what exactly is wrong? Could you share what you did and the screenshot?

[rochdev]

The steps I took:

• Create app.js and package.json with the above code in a new folder.
• Run yarn install in the folder.
• Run DD_TRACE_DEBUG=true node app

I get this output (there are more spans but this would be the relevant one):

[{"trace_id":"0d0383f7a9007798","span_id":"0d0383f7a9007798","parent_id":"0000000000000000","name":"mongodb.query","resource":"find myProject.documents {}","error":0,"meta":{"service":"mongodb-dev","env":"dev-sattarab","version":"dev-sattarab","runtime-id":"4b5db7f7-45fa-4e3d-b59d-d7933dd45bc4","span.kind":"client","db.name":"myProject.documents","mongodb.query":"{}","out.host":"127.0.0.1","out.port":"27017"},"metrics":{"_dd.agent_psr":1,"_dd.measured":1,"_sampling_priority_v1":1},"start":1651696583032001500,"duration":2357666,"service":"mongodb-dev-mongodb","type":"mongodb"}]

This is what I see in the UI:

Not the prettiest flame graph as queries would usually happen during a request, but I do get the trace. I tried with Node 14 as well to be sure, and it still works. Not sure what else could the issue. Can you reproduce in Docker maybe to remove anything specific to the environment?

[apank]

@sattarab - I'm having the same issue with mongoose after updating dd-trace.

For me if I use dd-trace 2.2.0 all my mongo traces are logged; but if I upgrade any higher the mongo traces stop being ingested.

[sattarab]

@rochdev It works when I downgrade it to 2.2 so I am going to do that for now. @apank thankyou so much for the suggestion

[guspio]

Hi!

With version 2.7.0, I have the APM traces of MongoDB again!

Thanks

[sattarab]

@guspio doesn't work with 2.7.0 for us

[apank]

@rochdev It works when I downgrade it to 2.2 so I am going to do that for now. @apank thankyou so much for the suggestion

np, it's very frustrating - with every release they break more things - there's other integrations that don't get pulled in anymore in the newer versions such as bunyan

[rochdev]

@apank Can you provide a reproduction? So far every reproduction that was provided was working as expected.

[nicolas-vivot]

Also it looks like the environment variable DD_TRACE_DISABLED_PLUGINS to specifically disabled some plugins is completely ignored by the newest versions.. How to suddenly explodes the counter of billed traces on Datadog with a small upgrade of a single library if you don't pay attention.. :/

[ondrejnebesky]

Bumped into this issue with [email protected] and [email protected] on NodeJS 16. Downgrade to 2.3.0 resolves the issue, but I would rather use the current version. I'll post simple case here when ready.

[sattarab]

MongoDB 4.7 doesn't work with version 2.2.0 or the latest version. Will post a workaround if I find anything

[sattarab]

@apank have you tried it with mongodb 4.7.0?

[apank]

I'm using mongoose v5.13.14 which use mongodb v3.7.3

I tested the latest version of dd-trace (2.10.0) - it didn't capture any of the mongo traces at all :(

[rochdev]

That could very well be a load order issue. Is the tracer initialized before any other module is imported? A good way to ensure this is to require it on the command line instead of in the process, for example with node -r dd-trace/init server.js, or if you need to programmatically configure the tracer then you would require your own initialization file instead with for example node -r ./datadog.js server.js. This can also be set with an environment variable if that's easier for your app with NODE_OPTIONS='-r dd-trace/init'.

If that doesn't work, we would need a reproduction snippet because I was never able to reproduce the issue with mongodb and our tests are all passing as well. I tried to break it on purpose and was still getting the spans, even with the above snippet.

[avivshafir]

wasn't working for me either, locked versions to "mongodb": "4.4.1", and "dd-trace": "2.5.0" then it started working.

[apank]

I use mongoose

On Thu, Jun 16, 2022, 11:07 AM sattarab @.***> wrote:

@apank https://github.com/apank have you tried it with mongodb 4.7.0?

— Reply to this email directly, view it on GitHub https://github.com/DataDog/dd-trace-js/issues/2006#issuecomment-1157770096, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTA7O7IZ6D36YBYZTNOBKDVPM7MBANCNFSM5UDGVPLQ . You are receiving this because you were mentioned.Message ID: @.***>

[apank]

@avivshafir @sattarab @guspio @ondrejnebesky @nicolas-vivot

Great news!

A few weeks ago I was able to try @rochdev's suggestion above and everything has been working as expected for me in v3.9.3 😎 😎 😎

I did the following - In my app I created a file called datadog.js in my app's src/config directory.

const tracer = require('dd-trace');
const { ENV } = process.env;

tracer.init({
  service: 'omega',
  version: 'alpha.beta.meta',
  logInjection: true,
  profiling: true,
  runtimeMetrics: true,
  startupLogs: true,
  appsec: true,
  env: ENV,
});
tracer.use('mongodb-core', {
  service: 'delta-db',
  measured: true,
});
// other plugin configs etc....

Then in package.json I updated the scripts object start property to first require the ./src/config/datadog.js. The -r cli arg in node ensures that the module is preloaded at startup.

"scripts": {
    "start": "node -r ./src/config/datadog.js src/index.js",
}

In my prior configuration; dd-tracer was being initialized in src/index.js like so below. When I would update dd-tracer to any version higher than v2.2.0 - mongo and other traces would not come through.

const express = require('express');
const tracer = require('dd-trace');
const routes = require('./routes/');
const { ENV } = process.env;

if (config.env === 'production') {
  tracer.init({
    service: 'omega',
    version: 'alpha.beta.meta',
    logInjection: true,
    profiling: true,
    runtimeMetrics: true,
    startupLogs: true,
    appsec: true,
    env: ENV,
  });
}

const app = express();

When I initially read the instrumentation docs (in hindsight, not too closely obviously 😛) - I believed since I was importing the library and initializing before initiating the express server itself that the above was valid; Most importantly, this setup worked as expected for v2.2.0 and below.

@rochdev - thank you very much for all your assistance and patience on this matter! ❤️

[rochdev]

@sattarab Do the same steps that worked for @apank also work for you?

[sattarab]

yes it works thankyou so much for this.