dd-trace plugin fails to resolve module due to missing `main` entrypoint on `standalone` builds

[lukexor]

Verify canary release

• [X] I verified that the issue exists in the latest Next.js canary release

Provide environment information

    Operating System:
      Platform: darwin
      Arch: x64
      Version: Darwin Kernel Version 20.6.0: Tue Jun 21 20:50:28 PDT 2022; root:xnu-7195.141.32~1/RELEASE_X86_64
    Binaries:
      Node: 16.10.0
      npm: 7.24.0
      Yarn: 1.22.19
      pnpm: N/A
    Relevant packages:
      next: 12.3.1-canary.5
      eslint-config-next: N/A
      react: 17.0.2
      react-dom: 17.0.2

warn  - Latest canary version not detected, detected: "12.3.1-canary.5", newest: "12.3.1".
        Please try the latest canary version (`npm install next@canary`) to confirm the issue still exists before creating a new issue.
        Read more - https://nextjs.org/docs/messages/opening-an-issue

What browser are you using? (if relevant)

N/A

How are you deploying your application? (if relevant)

next start in Docker

Describe the Bug

dd-trace provides a next plugin: https://github.com/DataDog/dd-trace-js/blob/master/docs/API.md#available-plugins

Next is built with output: "standalone" and deployed in a Docker container being started with node server.js.

Startup succeeds, but attempting to load localhost results in a module error:

info  - Loaded env from /app/.env
Listening on port 8080
Error: Cannot find module '/app/node_modules/next/dist/server/next.js'. Please verify that the package.json has a valid"main" entry
    at tryPackage (node:internal/modules/cjs/loader:353:19)
    at Function.Module._findPath (node:internal/modules/cjs/loader:566:18)
    at Module.Hook.Module.require (/app/node_modules/dd-trace/packages/dd-trace/src/ritm.js:109:26)
    at require (node:internal/modules/cjs/helpers:102:18)
    at parseCookie (/app/node_modules/next/dist/server/api-utils/index.js:19:43)
    at NodeNextRequest.get [as cookies] (/app/node_modules/next/dist/server/api-utils/index.js:129:27)
    at NodeNextRequest.get originalRequest [as originalRequest] (/app/node_modules/next/dist/server/base-http/node.js:16:34)
    at NextNodeServer.attachRequestMeta (/app/node_modules/next/dist/server/next-server.js:1299:46)
    at NextNodeServer.handleRequest (/app/node_modules/next/dist/server/base-server.js:132:18)
    at /app/node_modules/next/dist/server/next-server.js:817:20 {
  code: 'MODULE_NOT_FOUND',
  path: '/app/node_modules/next/package.json',
  requestPath: 'next'
}

Reason being that nexts package.json main entry points to a file that is not copied over in standalone mode - but the package.json itself is so dd-trace can't correctly import the package

Expected Behavior

Expected next page to be served correctly, instrumented with dd-trace

Link to reproduction

https://github.com/lukexor/nextjs-dd-trace-issue

To Reproduce

• npm i dd-trace
• Add a custom document: pages/_document.tsx with the default impl from the NextJS example: https://nextjs.org/docs/advanced-features/custom-document
• Add and init dd-trace at the top

import tracer from "dd-trace";

tracer.init();

• Update next.config.js to use output: "standalone"
• Add a Dockerfile:

# https://github.com/vercel/next.js/blob/canary/examples/with-docker/Dockerfile

FROM node:16.10.0-alpine
WORKDIR /app

COPY --chown=node:node next.config.js ./
COPY --chown=node:node public ./public/
COPY --chown=node:node .next/standalone/ ./
COPY --chown=node:node .next/static ./.next/static/

RUN apk update && \
  apk upgrade --no-cache && \
  rm -f /var/cache/apk/*.tar.gz

USER node

EXPOSE 8080
ENV PORT 8080

CMD [ "node", "server.js" ]

HEALTHCHECK --interval=15s --timeout=10s \
  CMD curl -f http://localhost:8080/ || exit 1

• npm run build
• docker build -f Dockerfile .
• docker run -p 80:8080 <image>
• Open http://localhost in your browser
• See the following message:

Listening on port 8080
Error: Cannot find module '/app/node_modules/next/dist/server/next.js'. Please verify that the package.json has a valid "main" entry
    at tryPackage (node:internal/modules/cjs/loader:353:19)
    at Function.Module._findPath (node:internal/modules/cjs/loader:566:18)
    at Module.Hook.Module.require (/app/node_modules/dd-trace/packages/dd-trace/src/ritm.js:109:26)
    at require (node:internal/modules/cjs/helpers:102:18)
    at parseCookie (/app/node_modules/next/dist/server/api-utils/index.js:19:43)
    at NodeNextRequest.get [as cookies] (/app/node_modules/next/dist/server/api-utils/index.js:129:27)
    at NodeNextRequest.get originalRequest [as originalRequest] (/app/node_modules/next/dist/server/base-http/node.js:16:34)
    at NextNodeServer.attachRequestMeta (/app/node_modules/next/dist/server/next-server.js:1315:46)
    at NextNodeServer.handleRequest (/app/node_modules/next/dist/server/base-server.js:133:18)
    at /app/node_modules/next/dist/server/next-server.js:833:20 {
  code: 'MODULE_NOT_FOUND',
  path: '/app/node_modules/next/package.json',
  requestPath: 'next'
}

[vikingair]

As a temporary workaround I created an additional task after the build was executed. Put as script into the package.json:

"build": "node scripts/build.mjs"

Create a file scripts/build.mjs with content:

import fs from 'fs';
import { execSync } from 'child_process';

execSync('./node_modules/.bin/next build', {
  env: { NODE_ENV: 'production', ...process.env },
  stdio: 'inherit',
});

// adjust the invalid "main" entry of the package.json of NextJS dependency to fix dd-trace package, which just checking
// its existence and failing otherwise
const nextJSPackageJsonFile = 'node_modules/next/package.json';
const nextJSPackageJson = JSON.parse(fs.readFileSync(nextJSPackageJsonFile, 'utf-8'));
nextJSPackageJson.main = './dist/server/next-server.js';
fs.writeFileSync(nextJSPackageJsonFile, JSON.stringify(nextJSPackageJson, null, 2));

Because dd-trace just wants to verify the existence of the package, you can modify the "main" entry to point to any just existing JS file. I have chosen above the './dist/server/next-server.js', but of course this can break again. You could also just create an empty JS file for that main entry if it doesn't exist. Might be cleaner.

[fotopixel]

The workaround works for me. But is it in general a bug? Because the module '/app/node_modules/next/dist/server/next.js' is not placed in the node_module after standalone build.

[vikingair]

I think the NextJS team was placing this file as entry point to the standalone server into the root of the standalone directory, but I would say that it is still a general bug, because the package.json becomes invalid by this change. If they would place at least as replacement an empty file for the main entry pointer that would be sufficient imo.

[raxityo]

My temporary solution:

# Dockerfile

# All the copy steps
COPY --from=builder x y

# After copy
# Change server/next.js → server/next-server.js in node_modules/next/package.json
RUN sed -i 's/server\/next.js/server\/next-server.js/' node_modules/next/package.json

I have made my deployment brittle to make sure it breaks loudly if dd-trace fails due to any external changes in nextjs.

[vikingair]

I've updated my workaround to the cleaner solution. Here I can share it once more:

import fs from 'fs';
import { execSync } from 'child_process';

execSync('./node_modules/.bin/next build', {
  env: { NODE_ENV: 'production', ...process.env },
  stdio: 'inherit',
});

const EMPTY_FILE_TEMPLATE = `"use strict";
Object.defineProperty(exports, "__esModule", {
    value: true
});
exports.default = void 0;
`;
const nextJSPackageDir = path.resolve('node_modules/next');
const nextJSPackageJson = JSON.parse(fs.readFileSync(path.join(nextJSPackageDir, 'package.json'), 'utf-8'));
const mainEntryFile = path.join(nextJSPackageDir, nextJSPackageJson.main);
if (!fs.existsSync(mainEntryFile)) fs.writeFileSync(mainEntryFile, EMPTY_FILE_TEMPLATE);

[itsmunim]

I am having a different problem altogether. I have dd-trace 3.8 as dependency.

This is my dockerfile-

# Installing dependencies
FROM node:16-alpine AS base

FROM base AS dependencies

ARG GITHUB_TOKEN

RUN apk add --no-cache libc6-compat vips-dev

WORKDIR /home/app
COPY package.json package-lock.json .npmrc ./

# prevent `husky install` triggered by `npm ci`
RUN npm set-script prepare ""
RUN npm config set '//npm.pkg.github.com/:_authToken' "${GITHUB_TOKEN}"
RUN npm ci

# Building the service
FROM base AS builder

ARG NODE_ENV
ARG SENTRY_AUTH_TOKEN
ENV NODE_ENV="${NODE_ENV}"

WORKDIR /home/app

COPY --from=dependencies /home/app/node_modules ./node_modules
RUN export NEXT_SHARP_PATH=./node_modules/sharp
COPY . .

ENV NEXT_TELEMETRY_DISABLED 1

RUN SENTRY_AUTH_TOKEN="${SENTRY_AUTH_TOKEN}" npm run build

# Service running container
FROM base AS runner
WORKDIR /home/app

ENV NEXT_TELEMETRY_DISABLED 1

RUN addgroup --system --gid 10001 nodejs
RUN adduser --system --uid 10001 nextjs

COPY --from=builder --chown=nextjs:nodejs /home/app/next.config.js ./
COPY --from=builder --chown=nextjs:nodejs /home/app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /home/app/.next/static ./.next/static
COPY --from=builder --chown=nextjs:nodejs /home/app/package.json ./package.json

USER nextjs

EXPOSE 3000

ENV PORT 3000

CMD ["node --require dd-trace/init", "./server.js"]

I keep on getting this error when running using docker-compose-

node:internal/modules/cjs/loader:988
   throw err;
   ^
 
 Error: Cannot find module '/home/app/node --require dd-trace/init'
     at Function.Module._resolveFilename (node:internal/modules/cjs/loader:985:15)
     at Function.Module._load (node:internal/modules/cjs/loader:833:27)
     at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
     at node:internal/main/run_main_module:22:47 {
   code: 'MODULE_NOT_FOUND',
   requireStack: []
 }

Even tried adding

COPY --from=dependencies --chown=nextjs:nodejs /home/app/node_modules/dd-trace ./node_modules/dd-trace

Right after COPY ... package.json line.

Still no luck. What might be causing this? Can anyone help?

Thanks!

[vikingair]

@dibosh Your issue is not related to this issue and should not have been raised here. If you're loading the dd-trace code via node --require in standalone mode then you are not making use of the standalone output resolution of NextJS anymore. You have to take care yourself that the used modules is copied into the standalone output before trying to load it.

So your issue will not be solved by solving this issue. As workaround for your issue:

Try adding the following import to your top-level _document.tsx:

import 'dd-trace';

Just by adding this line, you will tell NextJS to include the dependency in the standalone output.

[itsmunim]

@fdc-viktor-luft you are right! I am sorry for trying to piggyback, was not finding any relevant issue that mentions about the issues with datadog in standalone mode. Hence wanted to put it here.

Thanks for the suggestion - let me try what you suggested. If it does not work, I will create a new issue instead.

[tebuevd]

Hey folks! I was wondering if there's any point in loading dd-trace in a Vercel environment anyway? It is my impression that dd-trace requires the DataDog Agent installed in the environment. I even have an email from DD support that APM is not available in Vercel.

If that's the case, what are you using dd-trace for? @dibosh

[fotopixel]

@tebuevd i think the main discussion is about docker environment... that have nothing to do with Vercel.