dd-trace-js
dd-trace-js copied to clipboard
Modules are not instrumented (using webpack)
Hi there. I set up dd-trace according to the setup instructions and find that modules like http or express are not instrumented. I tried to find out the reason, digging in some of the dd-trace code and finding that e.g. express is not in the require.cache
because it only contains numeric values. Although I did not find anything in the instructions about it, I set the webpack optimization moduleIds
to named
. Now I see node module paths in require.cache
but still express is not there.
Is this a known problem? Am I doing somehting wrong here?
Thanks, Michael
Environment
- Operation system: Mac OSX
- Node version: 12.13.0
- Tracer version: 0.16.3
Alright, by now I figured out I had to externalize the node_modules using webpack-node-externals. Is this a required step in the setup? If it is, it should probably be mentioned in the setup instructions, no?
We could indeed recommend this module when using Webpack. It's possible that it won't be possible to use it in some setups, but I think it would work for most cases.
In all honesty, I didn't know about this module which is the why it's not currently documented, so thanks for the tip!
I'm not sure whether we're getting bitten by something similar - kinda a newb here - but we use babel to transpile our ES6 before shipping it. Would this be affecting us too - I realize, naturally, that we're not talking about WebPack, but I figure it can't hurt to ask.
Hi @armenr In our case the application and the node modules are bundled into a single JS file. The problem is that somehow the "require-in-the-middle" module which is used by dd-trace-js doesn't trigger if the node modules are bundled.
If you're just using babel to transpile ES6 to JS and are not bundling, you probably get bitten by something else...
@mruoss - I've used PKG, NCC, and NEXE - and have been bitten in ALL three cases by EXACTLY what you've been bitten by. I moved our entire setup (dev & prod) to Docker, JUST to get a "vanilla" set up, as an attempt to benefit from auto-instrumentation...but it doesn't look like it's instrumented fully.
"If you're just using babel to transpile ES6 to JS and are not bundling, you probably get bitten by something else..."
:) - How can I go about finding out?
:) - How can I go about finding out?
If you have no node_modules folder inside your docker image, this is your issue.
Alright, see the referenced issue above. Bundling node modules is not supported by the require-in-the-middle
module. As mentioned before, it might be helpful to mention this also here in the setup instructions somewhere.
@armenr Simply using Babel should not cause any problems, although you have to make sure to avoid hoisting, similar to what we recommend for TypeScript in the setup docs, otherwise auto-instrumentation will not work properly.
Thanks for the quick and helpful response, @rochdev! :)
We utilize an entrypoint file to fire up our NodeJS APIs in the following pattern/manner:
/* eslint-disable global-require */
const { join } = require('path');
require('dotenv').config({ path: join(__dirname, '.env') });
if (process.env.NODE_ENV === 'production') {
require('./dist/connectors/connectTracer.js');
require('./dist/server.js');
} else {
require('./src/connectors/connectTracer.js');
require('./src/server.js');
}
Is this sufficient/correct?
@armenr Assuming that the tracer is initialized synchronously in connectTracer.js
, then it looks correct. Which auto-instrumentation seems to be missing?
@rochdev --
synchronously, yes. --> see below
const tracer = require('dd-trace');
const {
DD_ENV,
DD_TRACE_DEBUG,
DD_SERVICE_NAME,
DD_TRACE_ENABLED,
DD_LOGS_INJECTION,
DD_PLUGINS_ENABLED,
DD_RUNTIME_METRICS_ENABLED,
} = process.env;
module.exports = tracer.init({
analytics: true,
debug: DD_TRACE_DEBUG,
enabled: DD_TRACE_ENABLED,
env: DD_ENV,
logInjection: DD_LOGS_INJECTION,
plugins: DD_PLUGINS_ENABLED,
runtimeMetrics: DD_RUNTIME_METRICS_ENABLED,
service: DD_SERVICE_NAME,
});
Alright, by now I figured out I had to externalize the node_modules using webpack-node-externals. Is this a required step in the setup? If it is, it should probably be mentioned in the setup instructions, no?
Hey @mruoss how could you fix that to use dd-trace in your webpack application? I've been falling in the same problem here a few days...
@williancolognesitrimble
my webpack config currently looks like this:
import nodeExternals from 'webpack-node-externals'
const config = {
entry: '...',
externals: [nodeExternals()],
... other_configuration
}
I've also encountered this problem with node10 and dd-trace while trying to bundle with webpack:
ERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/language/printer' in '/home/maschmann/git<package>/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js 10:18-53
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/signature.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/index.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/plugins/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/platform/node/index.js
@ ./node_modules/dd-trace/packages/dd-trace/index.js
@ ./node_modules/dd-trace/index.js
@ ./app.js
@ multi ./app.jsERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/language/visitor' in '/home/maschmann/git/<package>/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js 9:18-53
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/signature.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/index.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/plugins/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/platform/node/index.js
@ ./node_modules/dd-trace/packages/dd-trace/index.js
@ ./node_modules/dd-trace/index.js
@ ./app.js
@ multi ./app.jsERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/utilities' in '/home/maschmann/git/<package>/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js 11:20-48
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/signature.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/index.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/plugins/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/platform/node/index.js
@ ./node_modules/dd-trace/packages/dd-trace/index.js
@ ./node_modules/dd-trace/index.js
@ ./app.js
We also ran into this issue adding dd-trace
to our Angular app (Angular v9.1 universal using webpack). It looks like it could be a peer dependency issue on the plugins. We are going to try your approach @mruoss as a temporary workaround.
Let us know if you need more information @rochdev.
ERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/language/printer' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
ERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/language/visitor' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
ERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/utilities' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
ERROR in ./node_modules/spdx-expression-parse/scan.js
Module not found: Error: Can't resolve 'spdx-exceptions' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/spdx-expression-parse'
ERROR in ./node_modules/spdx-correct/index.js
Module not found: Error: Can't resolve 'spdx-license-ids' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/spdx-correct'
ERROR in ./node_modules/spdx-expression-parse/scan.js
Module not found: Error: Can't resolve 'spdx-license-ids' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/spdx-expression-parse'
ERROR in ./node_modules/spdx-expression-parse/scan.js
Module not found: Error: Can't resolve 'spdx-license-ids/deprecated' in '/Users/kfulgham/dev/flo/flosports-webapp/node_modules/spdx-expression-parse'
Until a fix is made, our team found a temporary workaround for Angular 9 CLI & webpack when using Angular Universal (server-side rendering).
Created an externals reference:
// webpack.config.js
module.exports = {
externals: [/^dd-trace/]
}
And added a customWebpackConfig
property to to our angular.json
in projects.our-project-name.architect.server
{
"builder": "@angular-builders/custom-webpack:server",
"options": {
"customWebpackConfig": {
"path": "./webpack.config.js"
},
"outputPath": "dist/flosports-webapp/server",
"main": "server.ts",
"tsConfig": "tsconfig.server.json",
"externalDependencies": ["@firebase/firestore"]
}
//...
}
We're running into the same issue as @kalebdf and @maschmann - webpack is failing for our server-side project. We're following the guidance here https://docs.datadoghq.com/tracing/setup/nodejs/
For graphql
, it needs to be set as an external in the Webpack config since it's only required when the plugin is loaded, which is not the case when graphql
is not used. This is because Webpack doesn't work with conditional requires which are used in that plugin.
For spdx-license-ids
, it's a transitive dependency from one of our dependencies (read-pkg
) and it should be included by Webpack, not sure why this isn't the case, I'll have to look into it.
Thanks for responding and for this package! I'll be blunt and ask, any update @rochdev ?
@amitizle Are you referring to the issue with spdx-license-ids
? Is this still an issue?
@rochdev yes it is.
@amitizle Ok, for some reason I assumed it was fixed. I'll look into it this week.
@amitizle I tried to take a quick look at the dependency tree but nothing stood out. I'm not super well versed Webpack wise, could you create a small replication snippet I could use as the basis of the investigation?
@rochdev me neither 😄 Let me try and write an entire setup that is similar to ours, I'll post in here with an update. Thanks a lot for the quick responses.
I think I found the issue with the spdx-* dependencies : it's because these packages only contain index.json files and this extension was not listed in my webpack.config.js. I solved my problem with :
module.exports = {
resolve: {
extensions: ['.js', '.jsx', '.ts', '.tsx', '.json'],
{...}
}
Hope it solves the problem for everyone!
I am using nestjs monorepo webpack build. works for me
const mergedOptions = merge(options, {
resolve: {
extensions: ['.js', '.ts', '.json'],
},
plugins: [
new webpack.IgnorePlugin({
checkResource(resource) {
const lazyImports = [
//...
'graphql/language/printer',
'graphql/language/visitor',
'graphql/utilities',
'spdx-exceptions',
'spdx-license-ids',
'spdx-license-ids/deprecated',
];
if (!lazyImports.includes(resource)) {
return false;
}
try {
require.resolve(resource, {
paths: [process.cwd()],
});
} catch (err) {
return true;
}
return false;
},
}),
],
});
We've removed the dependency on read-pkg
so the issue with spdx-license-ids
should not longer happen.
We had the same issue with dd-trace like @maschmann had. The solution was that we had to set externals for those graphql packages within the webpack config file.
externals: [
'graphql/language/visitor',
'graphql/language/printer',
'graphql/utilities'
],
This resolved the issue for us.
Hi team, we are having a simlilar issue with dd-trace for a large number of Lambda's which are running on NodeJS. They are created using Yarn 2 and webpack to a single index.js file. Because of the treeshaking and nature of the Yarn PnP we don't have any node_modules, so using webpack-node-externals isn't an option either - as we don't have any node_modules to copy in the zip we place on AWS Lambda.
In case a fix in the library or its dependencies isn't possible, is there any alternative workaround in existence?
@vanseverk There was a mention in another issue of webpack-pnp-externals, I would give that a try for now. We do plan to support Webpack and PnP officially eventually but not in the short term.