dd-trace-js

dd-trace-js copied to clipboard

Describe the bug We are running APM in our application and seeing a sawtooth pattern in RSS and heap that looks like this:

When we disable APM, we do not see this pattern; memory remains flat. Here are two environments behind the same load balancer (top one is APM disabled, bottom one is APM enabled):

Our APM configuration looks like this:

require('dd-trace').init({
  analytics: true,
  scope: 'async_hooks',
  trackAsyncScope: false
})

We are using the following plugins (automatically):

• aws_sdk
• express
• fs
• http
• mongodb_core
• mongoose
• net
• winston

We have tried some interventions:

1. removed scope: 'async_hooks' to allow the tracer to choose a more modern scope manager: this doesn't seem to have made a difference
2. turned off mongodb_core plugin: we can't tell if this is helping or not, so we will do more tests

We have added debug logging to track number of Promises, since that seems to have helped in https://github.com/DataDog/dd-trace-js/issues/759. After 48 hours with APM enabled:

• i-0e26b3b04e1e544ec Promises: 3312 (unresolved), 109 (current), 7201752 (created)
• i-0ed6974bec8387da1 Promises: 5750 (unresolved), 747 (current), 9432309 (created)

I would like to see if disabling APM in these environments shows the same high number of unresolved and current Promises. Stay tuned on that.

Finally, we have taken heap snapshots from a couple of environments; here is one showing a lot of heap being consumed by spans:

We had hoped to be able to open this issue with more conclusions and a simple reproduction, but we have had trouble reproducing this outside of our real customer environments. We have several customer environments that demonstrate the sawtooth pattern where we are able to perform experiments.

Environment

• Operating system: Debian Buster
• Node version: 14.17.5
• Tracer version: 1.3.0
• Agent version: 7.27.0

Hi @mhassan1, do you have any update on this issue? We're suspecting that we're also affected by this as we're seeing periodic OOM errors from our prod and our heap snapshots also show that they're consumed by spans. Happy to provide more details if it's helpful.

@taishi8117 Can you share the dependencies from your package.json? This would help isolating if the issue could come from a specific plugin.

Closing this issue for now since it's so old, a few major releases have happened, and we've fixed a few memory leaks since this issue was created. If it's still relevant please tag us and reopen, and also provide the requested package.json file.