DataDog
[BUG]: Requests and APM metrics zero-ing out with just a dd-trace upgrade 5.24.0 -> 5.35.0
Tracer Version(s)
5.35.0
Node.js Version(s)
22.11.0
Bug Report
For our Node application, we pin dd-trace just to the major version, specifically v5. As part of a routine package-lock.json regeneration, our dd-trace version was bumped from 5.24.0 to 5.35.0. Immediately after release, our Datadog monitors for low request rates went off, showing that requests and all APM metrics were zero-ing out. But by all other monitoring, the website powered by this application was fine. The application was clearly responding to requests, and all traffic was normal.
On closer inspection of the released pull requests, we noted the upgrade in dd-trace, and maybe more importantly a major version upgrade to dd-trace's dependency @datadog/native-metrics. We can't say for sure if that was the cause, but after rolling back these changes, all Datadog activity returned to normal. For now, we've pinned our dd-trace version exactly to 5.24.0.
Shown below is our requests, segmented by release:
light blue = dd-trace 5.24.0 dark blue = dd-trace 5.35.0 upgrade, requests fall off a cliff purple = back to dd-trace 5.24.0, requests return to normal
Reproduction Code
No response
Error Logs
No response
Tracer Config
tracer.init({
logInjection: true,
runtimeMetrics: true,
version: <release version>
});
Operating System
GNU/Linux
Bundling
No Bundling
Same problem, dd-trace 5.24.0 work fine, 5.31.0+ stop working, but with the plugin amqplib, basic.get resources.
Is this still an issue with newer versions (5.50.0 and above)? If yes, please reach out to our support https://www.datadoghq.com/support/. They will prioritize looking into this in that case! We can't sadly provide such support through github.
@BridgeAR So far, request rate seems stable. However all of our runtime metrics seem to have bottomed out, which is unusual for our releases, and we can't yet explain why we're seeing this.
@BridgeAR Here's an updated look at our Nodejs runtime metrics before and after upgrading our version of dd-trace from v5.24 to v5.51. Any idea why we'd be seeing such significant changes?
Same problem with update from 5.49.0 to 5.59.0. All custom metrics go to zero. I've opened a ticket for the datadog support team.
We're having the same problem in our team, we've been postponing the upgrade for several versions now, that we're currently stuck on 5.44.0 due to this. @antonio-masotti any luck with their support ?
