open-telemetry
wrong import for "import-in-the-middle"
What happened?
Steps to Reproduce
Import from "@opentelemetry/instrumentation" and bundle using esbuild with format "cjs"
Expected Result
The build should proceed without warning and not crash during runtime
Actual Result
The build produces an error and crashes during runtime
Additional Details
During build time the following output is shown
▲ [WARNING] Constructing "ImportInTheMiddle" will crash at run-time because it's an import namespace object, not a constructor [call-import-namespace]
../../.yarn/__virtual__/@opentelemetry-instrumentation-virtual-0402264686/0/cache/@opentelemetry-instrumentation-npm-0.40.0-a48dab8126-03755fc091.zip/node_modules/@opentelemetry/instrumentation/build/esm/platform/node/instrumentation.js:258:30:
258 │ var esmHook = new ImportInTheMiddle([module_2.name], { internals: false }, hookFn);
╵ ~~~~~~~~~~~~~~~~~
Consider changing "ImportInTheMiddle" to a default import instead:
../../.yarn/__virtual__/@opentelemetry-instrumentation-virtual-0402264686/0/cache/@opentelemetry-instrumentation-npm-0.40.0-a48dab8126-03755fc091.zip/node_modules/@opentelemetry/instrumentation/build/esm/platform/node/instrumentation.js:48:7:
48 │ import * as ImportInTheMiddle from 'import-in-the-middle';
│ ~~~~~~~~~~~~~~~~~~~~~~
OpenTelemetry Setup Code
import {
diag,
DiagLogLevel,
trace,
Tracer,
Span,
SpanKind,
SpanStatusCode,
} from "@opentelemetry/api";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { AwsLambdaInstrumentation } from "@opentelemetry/instrumentation-aws-lambda";
import { DnsInstrumentation } from "@opentelemetry/instrumentation-dns";
import { AwsInstrumentation } from "@opentelemetry/instrumentation-aws-sdk";
import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";
import { AWSXRayIdGenerator } from "@opentelemetry/id-generator-aws-xray";
import { AWSXRayPropagator } from "@opentelemetry/propagator-aws-xray";
import {
SemanticAttributes,
SemanticResourceAttributes,
CloudProviderValues,
CloudPlatformValues,
} from "@opentelemetry/semantic-conventions";
try {
diag.setLogger(logger as any, DiagLogLevel.ALL);
const provider = new NodeTracerProvider({
idGenerator: new AWSXRayIdGenerator(),
});
const spanProcessor = new BatchSpanProcessor(
new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
})
);
provider.addSpanProcessor(spanProcessor);
provider.register({
propagator: new AWSXRayPropagator(),
});
trace.setGlobalTracerProvider(provider);
registerInstrumentations({
instrumentations: [
new AwsInstrumentation({}),
new AwsLambdaInstrumentation({
requestHook: (span, { event, context }) => {
const [, , , , accountId] = context.invokedFunctionArn.split(":");
const functionArn = context.invokedFunctionArn
.split(":", 7)
.join(":");
span.setAttributes({
[SemanticAttributes.AWS_LAMBDA_INVOKED_ARN]:
context.invokedFunctionArn,
[SemanticResourceAttributes.AWS_LOG_GROUP_NAMES]: [
context.logGroupName,
],
[SemanticResourceAttributes.AWS_LOG_STREAM_NAMES]: [
context.logStreamName,
],
[SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]:
process.env.ENV_TIER,
[SemanticResourceAttributes.CLOUD_ACCOUNT_ID]: accountId,
[SemanticResourceAttributes.CLOUD_PROVIDER]:
CloudProviderValues.AWS,
[SemanticResourceAttributes.CLOUD_REGION]: process.env.AWS_REGION,
[SemanticResourceAttributes.CLOUD_PLATFORM]:
CloudPlatformValues.AWS_LAMBDA,
"cloud.resource_id": functionArn,
// https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/faas.md
//TODO infer `faas.trigger`
//TODO infer `faas.document.*`
[SemanticResourceAttributes.FAAS_NAME]: context.functionName,
[SemanticAttributes.FAAS_EXECUTION]: context.awsRequestId,
[SemanticResourceAttributes.FAAS_ID]: functionArn,
[SemanticResourceAttributes.FAAS_INSTANCE]: context.logStreamName,
[SemanticResourceAttributes.FAAS_VERSION]:
context.functionVersion,
[SemanticResourceAttributes.FAAS_MAX_MEMORY]:
Number(context.memoryLimitInMB) * 1024 * 1024,
[SemanticResourceAttributes.HOST_ARCH]: os.arch(),
[SemanticResourceAttributes.SERVICE_NAME]:
process.env.WARRIFY_SERVICE,
[SemanticResourceAttributes.SERVICE_NAMESPACE]:
process.env.WARRIFY_NAMESPACE,
});
},
responseHook: (span, { err, res }) => {
if (err instanceof Error) {
span.recordException(err);
}
},
}),
new DnsInstrumentation({}),
new HttpInstrumentation(),
],
});
} catch (err: any) {
logger.error(err);
}
const tracer = trace.getTracer(
process.env.AWS_LAMBDA_FUNCTION_NAME ?? "unknown"
);
package.json
{
"name": "@warrify/lambda-middlewares",
"version": "0.3.10",
"description": "",
"keywords": [],
"homepage": "",
"license": "ISC",
"main": "dist/cjs/index.js",
"module": "dist/esm/index.js",
"types": "dist/esm/index.d.ts",
"exports": {
".": {
"import": "./dist/esm/index.js",
"require": "./dist/cjs/index.js"
},
"./*": {
"import": "./dist/esm/*.js",
"require": "./dist/cjs/*.js"
}
},
"dependencies": {
"@aws-sdk/client-dynamodb": "^3.360.0",
"@aws-sdk/client-secrets-manager": "^3.360.0",
"@aws-sdk/client-servicediscovery": "^3.360.0",
"@aws-sdk/client-ssm": "^3.360.0",
"@aws-sdk/client-sts": "^3.360.0",
"@aws-sdk/lib-dynamodb": "^3.360.0",
"@aws-sdk/node-http-handler": "^3.360.0",
"@aws-sdk/util-dynamodb": "^3.360.0",
"@middy/cloudwatch-metrics": "^4.5.2",
"@middy/core": "^4.5.2",
"@middy/do-not-wait-for-empty-event-loop": "^4.5.2",
"@middy/error-logger": "^4.5.2",
"@middy/event-normalizer": "^4.5.2",
"@middy/http-content-negotiation": "^4.5.2",
"@middy/http-cors": "^4.5.2",
"@middy/http-error-handler": "^4.5.2",
"@middy/http-event-normalizer": "^4.5.2",
"@middy/http-header-normalizer": "^4.5.2",
"@middy/http-json-body-parser": "^4.5.2",
"@middy/http-multipart-body-parser": "^4.5.2",
"@middy/http-partial-response": "^4.5.2",
"@middy/http-response-serializer": "^4.5.2",
"@middy/http-security-headers": "^4.5.2",
"@middy/http-urlencode-body-parser": "^4.5.2",
"@middy/http-urlencode-path-parser": "^4.5.2",
"@middy/input-output-logger": "^4.5.2",
"@middy/secrets-manager": "^4.5.2",
"@middy/sqs-partial-batch-failure": "^4.5.2",
"@middy/ssm": "^4.5.2",
"@middy/sts": "^4.5.2",
"@middy/util": "^4.5.2",
"@middy/validator": "^4.5.2",
"@opentelemetry/api": "^1.4.1",
"@opentelemetry/exporter-trace-otlp-http": "^0.40.0",
"@opentelemetry/id-generator-aws-xray": "^1.1.2",
"@opentelemetry/instrumentation": "^0.40.0",
"@opentelemetry/instrumentation-aws-lambda": "^0.35.3",
"@opentelemetry/instrumentation-aws-sdk": "^0.34.3",
"@opentelemetry/instrumentation-dns": "^0.31.5",
"@opentelemetry/instrumentation-http": "^0.40.0",
"@opentelemetry/propagator-aws-xray": "^1.2.1",
"@opentelemetry/sdk-node": "^0.40.0",
"@opentelemetry/sdk-trace-base": "^1.14.0",
"@opentelemetry/sdk-trace-node": "^1.14.0",
"@opentelemetry/semantic-conventions": "^1.14.0",
"@types/aws-lambda": "^8.10.119",
"ajv": "^8.12.0",
"ajv-errors": "^3.0.0",
"ajv-formats": "^2.1.1",
"ajv-formats-draft2019": "^1.6.1",
"aws-embedded-metrics": "^4.1.0",
"aws-lambda": "^1.0.7",
"aws-xray-sdk-core": "^3.5.0",
"date-fns": "^2.30.0",
"dd-trace": "^4.3.0",
"decimal.js": "^10.4.3",
"graphql": "^16.7.1",
"http-status-codes": "^2.2.0",
"joi": "^17.9.2",
"lodash": "^4.17.21",
"src": "link:./",
}
"files": [
"dist/**/*.d.ts",
"dist/**/*.[tj]s",
"dist/**/*.[tj]sx",
"dist/**/*.json"
]
}
Relevant log output
Our Lambda functions are instrumented with datadog, hence the stack trace shows datadog files all the way
{
"errorType": "Runtime.ImportModuleError",
"errorMessage": "Error: Cannot find module 'is-core-module/package.json'\nRequire stack:\n- /var/task/index.js\n- /opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js\n- /opt/nodejs/node_modules/datadog-lambda-js/runtime/index.js",
"stack": [
"Runtime.ImportModuleError: Error: Cannot find module 'is-core-module/package.json'",
"Require stack:",
"- /var/task/index.js",
"- /opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js",
"- /opt/nodejs/node_modules/datadog-lambda-js/runtime/index.js",
" at ImportModuleError.ExtendedError [as constructor] (/opt/nodejs/node_modules/datadog-lambda-js/runtime/errors.js:113:28)",
" at new ImportModuleError (/opt/nodejs/node_modules/datadog-lambda-js/runtime/errors.js:123:42)",
" at /opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:273:31",
" at step (/opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:43:23)",
" at Object.throw (/opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:24:53)",
" at rejected (/opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:16:65)"
]
}
We're having the same problem over at https://github.com/highlight/highlight
It just kicked up when we started consuming the CJS version instead of the ESM.
I made a quick edit to @opentelemetry/instrumentation/build/esm/platform/node/instrumentation.ts:23.
I changed the import to import ImportInTheMiddle from 'import-in-the-middle';.
It worked great.
@deltaepsilon @matthias-pichler-warrify I think the latest release should have resolved this
@deltaepsilon @matthias-pichler-warrify I think the latest release should have resolved this
Awesome! I'm working on this now. I'll update and report back.
Oh I'm sorry I conflated this with a different issue. @trentm do you have any idea what's going on here? I vaguely remember a similar problem with is-core-module in the past.
I'll make some guesses, but I'm not that confident with either esbuild bundling, or with the "build/esm/..." output of OTel packages.
I vaguely remember a similar problem with is-core-module in the past.
is-core-module is imported by resolve which is imported by require-in-the-middle. Ah, the "similar problem in the past" was perhaps with require-in-the-middle, rather than import-in-the-middle. For the earlier case, IIRC, there was this discussion: https://github.com/browserify/resolve/issues/297
(and these two linked issues: elastic/require-in-the-middle#71 open-telemetry/opentelemetry-js#3813)
I think that was resolved when [email protected] was marked as the "latest" dist-tag -- rather than the [email protected] release which was reaching into the is-core-module/... installation.
I'm not sure that is necessarily the same issue as here. If it is the same issue, then I think you would be broken if npm ls -a resolve shows that [email protected] is being used in your install/build.
If someone hitting this error created a full repro with the specific commands to run, that would help debug this.
Oh, actually this looks more similar: https://github.com/open-telemetry/opentelemetry-js/issues/3701 (Also any of these matching issues: https://github.com/open-telemetry/opentelemetry-js/issues?q=+%22import+namespace+object%22+)
The fix/workaround then was to:
- Get a new version of
require-in-the-middlethat provided a named export for theHookfunction -- rather than just being a default export: https://github.com/open-telemetry/opentelemetry-js/issues/3701#issuecomment-1488918332 - Then updating the OTel usage to use the named export: https://github.com/open-telemetry/opentelemetry-js/pull/3727/files#diff-c2b4fb5752545bff41e0e0eabf5af35a41ef65c7fb0dec41c258690bdb71e7f9R18
To be honest I get a little lost in the "TypeScript import syntax" -> "commonjs and/or esm built code" -> "use of that code in require() or import ... or import()" -> "possibly bundled by one of many bundlers". However, if this is the same thing (I wasn't able to repro last time), then we'd need/want to do the same changes to import-in-the-middle.
@trentm I feel your pain regarding ESM/CJS/Whatever-else-they've-come-up-with.
I solved the problem for my local environment by editing the import deep in down in node_modules. I found import * as ImportInTheMiddle from 'import-in-the-middle'; and changed it to import ImportInTheMiddle from 'import-in-the-middle';.
That was all the bundler needed to be happy again.
Of course, the moment I send this to our CI pipeline it'll install the package from scratch.
Running into this same issue trying to use opentelemetry in a project built using esbuild. If it helps anyone I ended up working around the issue in a semi-durable way using https://github.com/ds300/patch-package to make the change described above.
Would be awesome to have a "real" fix and not require the hackaround
@pichlermarc - Is there a reason why this: https://github.com/open-telemetry/opentelemetry-js/blob/20182d8804f0742ddb1b2543ad9de0d88a941a65/experimental/packages/opentelemetry-instrumentation/src/platform/node/instrumentation.ts#L28
Was not updated to this:
import { ImportInTheMiddle } from 'import-in-the-middle';
In the latest release?
Because at runtime this will absolutely lead to an error being thrown for ESM based applications.
Already posted in a proposed fix PR but I wanted to put this here too to get more eyes on it:
We're having a hard time validating this fix because we don't have a reproducer. Does anyone have a repro we can use? I'm also not convinced that after we fix this error your problems will be solved. To my knowledge, bundlers work by removing import/require statements and inlining code. Our instrumentations rely on the functioning of those statements so it may turn out that once this problem is fixed the instrumentations may not work anyway.
The obvious thing to do would be to fix the exports in import-in-the-middle and release a V2 that doesn't have invalid exports!
This is 100% what I would prefer. We also talked about changing the web instrumentations to not use the same instrumentation base class as node instrumentations because they work differently (no require/import capture), which would also resolve this issue at least in web.