Instrumentation files not included in standalone output

[moshie]

Link to the code that reproduces this issue

https://github.com/moshie/nextjs-issue

To Reproduce

1. Pull down the repo: https://github.com/moshie/nextjs-issue
2. run npm install
3. run npm run build
4. See in the .next/standalone/node_modules folder that the @splunk/otel prebuilds folder is not included

Current vs. Expected behavior

When using the @splunk/otel package in the instrumentation file the standalone output doesn't include the prebuilds from: node_modules/@splunk/otel/prebuilds/linux-x64/@splunk+otel.abi115.node

That results in this error:

Since outputFileTracingIncludes seems like it's tied to the pages and app directories I am not sure how best to include this file into my build?

I've tried in my docker file to include something similar to this:

COPY --chown=nonroot --from=builder /opt/application/node_modules/@splunk ./node_modules/@splunk
COPY --chown=nonroot --from=builder /opt/application/node_modules/.pnpm/@[email protected]_@[email protected] ./node_modules/.pnpm/@[email protected]_@[email protected]

But this isn't a very elegant solution since the path names contain versions and this would need to be updated everytime we update the @splunk/otel package.

I suppose this might have solved my issue but it's been removed (I don't know if it copied whole folders?):

unstable_includeFiles: ['node_modules/@splunk/otel'],

Provide environment information

Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000
  Available memory (MB): 32768
  Available CPU cores: 10
Binaries:
  Node: 20.16.0
  npm: 10.8.1
  Yarn: 1.22.22
  pnpm: 9.7.0
Relevant Packages:
  next: 14.2.5
  eslint-config-next: N/A
  react: 18.3.1
  react-dom: 18.3.1
  typescript: 5.5.4
Next.js Config:
  output: standalone

Which area(s) are affected? (Select all that apply)

Instrumentation, Output (export/standalone)

Which stage(s) are affected? (Select all that apply)

next build (local), next start (local)

Additional context

My app is dockerised and is using distoless. I thought about just running pnpm install @splunk/otel directly in the standalone folder but that didn't seem like it would work with distroless and also it's not very elegant either.

[admmasters]

There do seem to be some painpoints with nextjs and certain packages when utilising standalone - wondering if there is scope for more work on the mechanism which determines what to include in the bundle and how that works with binaries etc

[guinanlin]

via https://github.com/vercel/next.js/issues/49897 we can get the Instrumentation not working well with runtime=nodejs

after my testing on my 14.2.5 next.js version.

it will work well while pnpm run dev but not occur while runtime=nodejs, on pnpm start

[moshie]

Any update on this?

Edit by maintainer bot: Comment was automatically minimized because it was considered unhelpful. (If you think this was by mistake, let us know). Please only comment if it adds context to the issue. If you want to express that you have the same problem, use the upvote 👍 on the issue description or subscribe to the issue for updates. Thanks!

[zacharyblasczyk]

Addressing the OP

Sorry if this is verbose, but I am having a very similar issue for weeks and also can't seem to figure this out.

My issue seems to be related, but in general it seems like external otel integrations aren't well supported in containerized builds using the standalone output.

Specifically on the point of including the dependencies in the final standalone build, I think you need to add a serverComponentsExternalPackages block to your experimental section of the next.config.js like this:

  experimental: {
    instrumentationHook: true,
    optimizePackageImports: ["bullmq", "googleapis"],
    /** @see https://github.com/open-telemetry/opentelemetry-js/issues/4297 */
    serverComponentsExternalPackages: [
      "@opentelemetry/sdk-node",
      "@opentelemetry/auto-instrumentations-node",
      "@opentelemetry/exporter-trace-otlp-http",
      "@appsignal/opentelemetry-instrumentation-bullmq",
      "@opentelemetry/instrumentation-pg",
      "@opentelemetry/resources",
      "@opentelemetry/semantic-conventions",
    ],
  },

In next 15 I think this has changed to serverExternalPackages re this.

You also need to set the ENV NODE_PATH in your dockerfile so that NODE_OPTIONS can work and node knows where to look for the node_modules folder.

i.e.

ENV NODE_PATH=/app/apps/webservice/node_modules
NODE_OPTIONS=--require=@opentelemetry/auto-instrumentations-node

[zacharyblasczyk]

It is also worth calling out that according to the docs:

The instrumentation file should be in the root of your project and not inside the app or pages directory. If you're using the src folder, then place the file inside src alongside pages and app.

[zacharyblasczyk]

Again, sorry for the long post. Please read. 🙏

On the point of a difference between local and containerized builds.

In my case—if anyone is willing to help me out—everything works fine locally, but as soon as I containerize the app (which is happening with standalone output using the above "solution") I lose a bunch of my traces.

The standalone folder and node server.js doesn't seem to have a way to initialize the register() function properly with all of its dependencies.

Specifically in my case, I lose the pg and pg.pool traces from: @opentelemetry/instrumentation-pg

Locally running pnpm dev:

@ctrlplane/webservice:dev: @vercel/otel: Configure propagator: tracecontext
@ctrlplane/webservice:dev: @vercel/otel: Configure propagator: baggage
@ctrlplane/webservice:dev: @vercel/otel: Configure sampler:  parentbased_always_on
@ctrlplane/webservice:dev: @vercel/otel: Configure trace exporter:  http/protobuf http://localhost:4318/v1/traces headers: <none>
@ctrlplane/webservice:dev: @vercel/otel/otlp: onInit
@ctrlplane/webservice:dev: @opentelemetry/api: Registered a global for trace v1.9.0.
@ctrlplane/webservice:dev: @opentelemetry/api: Registered a global for context v1.9.0.
@ctrlplane/webservice:dev: @opentelemetry/api: Registered a global for propagation v1.9.0.
@ctrlplane/webservice:dev: @vercel/otel: Configure instrumentations: fetch undefined
@ctrlplane/webservice:dev: @vercel/otel: started ctrlplane/webservice nodejs
@ctrlplane/webservice:dev: Fo found resource. r {
@ctrlplane/webservice:dev:   _attributes: {},
@ctrlplane/webservice:dev:   asyncAttributesPending: false,
@ctrlplane/webservice:dev:   _syncAttributes: {},
@ctrlplane/webservice:dev:   _asyncAttributesPromise: undefined
@ctrlplane/webservice:dev: }
...
# I see this further in the logs: 
@ctrlplane/webservice:dev:     instrumentationLibrary: {
@ctrlplane/webservice:dev:       name: '@opentelemetry/instrumentation-pg',
@ctrlplane/webservice:dev:       version: '0.47.1',
@ctrlplane/webservice:dev:       schemaUrl: undefined
@ctrlplane/webservice:dev:     },

As soon as I containerize and try and running it, some http traces still work, but I lose all postgres tracing.

Here is part my dockerfile:

...
RUN turbo build --filter=...@ctrlplane/webservice

FROM base AS runner
WORKDIR /app

COPY --from=installer --chown=nodejs:nodejs /app/apps/webservice/.next/standalone ./
COPY --from=installer --chown=nodejs:nodejs /app/apps/webservice/.next/static ./apps/webservice/.next/static
COPY --from=installer --chown=nodejs:nodejs /app/apps/webservice/public ./apps/webservice/public

EXPOSE 3000

ENV PORT=3000
ENV AUTH_TRUST_HOST=true
ENV NODE_ENV=production
ENV NODE_PATH=/app/apps/webservice/node_modules
ENV HOSTNAME=0.0.0.0

CMD ["node", "apps/webservice/server.js"]

I have tried two branches of tactics in a variety of ways that I want to share if it is helpful to debugging this.

1 Using instrumentation.ts only with @vercel/otel.

import type { PgInstrumentationConfig } from "@opentelemetry/instrumentation-pg";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { PgInstrumentation } from "@opentelemetry/instrumentation-pg";
import { WinstonInstrumentation } from "@opentelemetry/instrumentation-winston";
import { registerOTel } from "@vercel/otel";

export function register() {
  if (process.env.NEXT_RUNTIME === "nodejs") {
    const pgTraceConfig: PgInstrumentationConfig = {
      addSqlCommenterCommentToQueries: true,
      enhancedDatabaseReporting: true,
    };
    registerInstrumentations({
      instrumentations: [
        new PgInstrumentation(pgTraceConfig),
        new WinstonInstrumentation({
          logHook: (_, record) => {
            record["resource.service.name"] = "ctrlplane/webservice";
          },
        }),
      ],
    });
    registerOTel({
      serviceName: "ctrlplane/webservice",
      traceExporter: "auto",
    });
  }
}

2 Using a simple instrumentation.ts and instrumentation-node.ts with @opentelemetry/auto-instrumentations-node:

instrumentation.ts

export async function register() {
  if (process.env.NEXT_RUNTIME === "nodejs")
    await import("./instrumentation-node");
}

instrumentation-node.ts

import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { Resource } from "@opentelemetry/resources";
import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs";
import { NodeSDK } from "@opentelemetry/sdk-node";
import {
  AlwaysOnSampler,
  SimpleSpanProcessor,
} from "@opentelemetry/sdk-trace-base";
import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: "ctrlplane/webservice",
  }),
  spanProcessors: [new SimpleSpanProcessor(new OTLPTraceExporter())],
  logRecordProcessors: [new BatchLogRecordProcessor(new OTLPLogExporter())],
  instrumentations: [
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-fs": {
        enabled: false,
      },
      "@opentelemetry/instrumentation-net": {
        enabled: false,
      },
      "@opentelemetry/instrumentation-dns": {
        enabled: false,
      },
      "@opentelemetry/instrumentation-http": {
        enabled: true,
      },
      "@opentelemetry/instrumentation-pg": {
        enabled: true,
        enhancedDatabaseReporting: true,
        addSqlCommenterCommentToQueries: true,
      },
      "@opentelemetry/instrumentation-ioredis": {
        enabled: true,
      },
      "@opentelemetry/instrumentation-winston": {
        enabled: false,
      },
    }),
  ],
  sampler: new AlwaysOnSampler(),
});

try {
  sdk.start();
  console.log("Tracing initialized test");
} catch (error) {
  console.error("Error initializing tracing", error);
}

Conclusion

Both path 1 and 2 work fine locally, but when containerized, pg traces don't show and there is nothing clear in the the verbose debug logs to help determine why 😓

I have tried setting:

NEXT_OTEL_VERBOSE: 1
OTEL_LOG_LEVEL: debug
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: "http/protobuf"

to no avail.

[zacharyblasczyk]

Very similar to these issues:

https://github.com/backstage/backstage/issues/22555 https://github.com/vercel/next.js/issues/49897#issuecomment-2022510320

[tomups]

I ended up installing the missing packages during docker build and patching server.js to start instrumentation.

Extremely ugly, but it works...

Here is an example Dockerfile based on https://github.com/vercel/next.js/blob/canary/examples/with-docker/Dockerfile with the changes needed (they are towards the end of the file)

# syntax=docker.io/docker/dockerfile:1

FROM node:18-alpine AS base

# Install dependencies only when needed
FROM base AS deps
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
WORKDIR /app

# Install dependencies based on the preferred package manager
COPY package.json yarn.lock* package-lock.json* pnpm-lock.yaml* .npmrc* ./
RUN \
  if [ -f yarn.lock ]; then yarn --frozen-lockfile; \
  elif [ -f package-lock.json ]; then npm ci; \
  elif [ -f pnpm-lock.yaml ]; then corepack enable pnpm && pnpm i --frozen-lockfile; \
  else echo "Lockfile not found." && exit 1; \
  fi


# Rebuild the source code only when needed
FROM base AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .

# Next.js collects completely anonymous telemetry data about general usage.
# Learn more here: https://nextjs.org/telemetry
# Uncomment the following line in case you want to disable telemetry during the build.
# ENV NEXT_TELEMETRY_DISABLED=1

RUN \
  if [ -f yarn.lock ]; then yarn run build; \
  elif [ -f package-lock.json ]; then npm run build; \
  elif [ -f pnpm-lock.yaml ]; then corepack enable pnpm && pnpm run build; \
  else echo "Lockfile not found." && exit 1; \
  fi

# Production image, copy all the files and run next
FROM base AS runner
WORKDIR /app

ENV NODE_ENV=production
# Uncomment the following line in case you want to disable telemetry during runtime.
# ENV NEXT_TELEMETRY_DISABLED=1

RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs

COPY --from=builder /app/public ./public

# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static

# ----- NEW CODE FOR OTEL -----

# Needed to instal dependencies and to prevent 
# "failed to read machine uuid: Failed to open "/var/lib/dbus/machine-id": No such file or directory" 
# at runtime due to Alpine image
RUN apk add --no-cache git dbus

# Install otel
RUN npm install @vercel/otel @opentelemetry/sdk-logs @opentelemetry/api-logs @opentelemetry/instrumentation

# Copy instrumentation config
COPY --from=builder /app/src/instrumentation.js ./instrumentation.js

# Patch server.js so it starts instrumentation
RUN sed -i '1i\import { register } from "./instrumentation.js";\nregister();\n' server.js

# ENV OTEL_LOG_LEVEL="debug"
ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://[otel collector endpoint]:4318/"

# ----- END NEW CODE FOR OTEL -----

USER nextjs

EXPOSE 3000

ENV PORT=3000

# server.js is created by next build from the standalone output
# https://nextjs.org/docs/pages/api-reference/config/next-config-js/output
ENV HOSTNAME="0.0.0.0"
CMD ["node", "server.js"]

It seems with Next.js there is always some patching involved when you want to have proper insights into non-Vercel deploys :( see https://www.tomups.com/posts/log-nextjs-request-response-as-json/

[healqq]

Instrumentation (including pg) did work for us in Next 14, but with Next 15 we're seeing an issue similar to what @zacharyblasczyk described

[stephenliang]

You can notify the build trace output of these instrumentation libraries by importing them in instrumentation.ts. In my case, I was missing the @opentelemetry/auto-instrumentations-node-/register script. Adding an import will cause the build trace to include the file. If you do it this way, there will not be a need to do custom copy logic as it'd be part of the build trace output.

if (process.env.NEXT_RUNTIME === 'nodejs') {
  await import('./instrumentation.node');
  await import('@opentelemetry/auto-instrumentations-node/register');
}

[benedfit]

I managed to force @splunk/otel to output in the standalone build by adding the following to next.config.js:

outputFileTracingIncludes: {
  '*': ['./node_modules/@splunk/otel/**'],
},

There may be a slightly less eager selector that '*' but it worked for the purpose of ensuring the dependency is output

[carlosmaranje]

Edit

I solved this by moving the instrumentation.ts file inside the src folder. We now have otel traces. I have to check metrics. But so far so good.

---

I'm struggling with this too... My team is using turbo for the prod build. I can confirm the instrumentation file is being copied over inside the image, but it's being deliberately ignored by the compiler. I also added this to next.config.mjs

import { sharedConfig } from 'next-config';

/** @type {import('next').NextConfig} */
const nextConfig = {
    ...sharedConfig,
    env: {
        NEXT_PUBLIC_DEFAULT_LANG: process.env.NEXT_PUBLIC_DEFAULT_LANG,
    },
    outputFileTracingIncludes: {
        '/': [
            './instrumentation.ts',
            './node_modules/@vercel/otel/**',
            './node_modules/@opentelemetry/**',
        ],
    },
};

export default nextConfig;

The content of my instrumentation file is the same as in the otel guide:

import { registerOTel } from '@vercel/otel'

export async function register() {
  registerOTel({ serviceName: 'next-app' })
}

And this is my team's Dockerfile:

FROM node:22-alpine AS base

# ---
FROM base AS builder

RUN npm install -g [email protected]
RUN corepack enable && corepack prepare [email protected] --activate

ENV PNPM_HOME="/pnpm"
ENV PATH="$PNPM_HOME:$PATH"

RUN apk add --no-cache libc6-compat
RUN apk update
WORKDIR /app
RUN pnpm -g add [email protected]
COPY . .
RUN turbo prune --scope=hub --docker

# ---
FROM base AS installer
RUN npm install -g pnpm
RUN apk add --no-cache libc6-compat
RUN apk update
WORKDIR /app

COPY .gitignore .gitignore
COPY --from=builder /app/out/json/ .
COPY --from=builder /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
ENV HUSKY=0
RUN pnpm install --ignore-scripts

COPY --from=builder /app/out/full/ .
COPY turbo.json turbo.json

COPY --from=builder /app/apps/hub/instrumentation.ts ./apps/hub/instrumentation.ts

RUN pnpm turbo run build --filter=hub...

# ---
FROM base AS runner
WORKDIR /app

COPY --from=installer /app/apps/hub/next.config.mjs .
COPY --from=installer /app/apps/hub/package.json .


COPY --from=installer --chown=nextjs:nodejs /app/apps/hub/.next/standalone ./
COPY --from=installer --chown=nextjs:nodejs /app/apps/hub/.next/static ./apps/hub/.next/static
COPY --from=installer --chown=nextjs:nodejs /app/apps/hub/public ./apps/hub/public
RUN chmod 777 /app/apps/hub/.next

RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs

USER nextjs

EXPOSE 3000

CMD node apps/hub/server.js

Now, for debugging I did something I removed the runner step and built only until the installer just to discard that turbo prune might be removing them. I entered into the container and this is the folder structure:

/app
├── apps/
│   └── hub/
│       ├── .next/
│       │   ├── cache/
│       │   │   └── ...
│       │   ├── diagnostics/
│       │   │   └── ...
│       │   ├── server/
│       │   │   └── ...
│       │   ├── standalone/
│       │   │   ├── apps/
│       │   │   │   └── hub/
│       │   │   │       ├── src/
│       │   │   │       │   └── ...
│       │   │   │       ├── instrumentation.ts <----- The file is there after pnpm turbo run build is applied
│       │   │   │       ├── package.json
│       │   │   │       └── server.js
│       │   ├── static/
│       │   │   └── ...
│       │   ├── trace/
│       │   │   └── ...
│       │   ├── types/
│       │   │   └── ...
│       │   ├── BUILD_ID
│       │   ├── app-build-manifest.json
│       │   ├── app-path-routes-manifest.json
│       │   ├── build-manifest.json
│       │   ├── export-marker.json
│       │   ├── images-manifest.json
│       │   ├── next-minimal-server.js.nft.json
│       │   ├── next-server.js.nft.json
│       │   ├── package.json
│       │   ├── prerender-manifest.json
│       │   ├── react-loadable-manifest.json
│       │   ├── required-server-files.json
│       │   └── routes-manifest.json
│       ├── node_modules/
│       │   └── ...
│       ├── public/
│       │   └── ...
│       ├── src/
│       │   └── ...
│       ├── Dockerfile
│       ├── README.md
│       ├── build.sh
│       ├── instrumentation.ts  <----- The file is there after pnpm turbo run build is applied
│       ├── next-config.mjs
│       ├── next-env.d.ts
│       ├── package.json
│       ├── postcss.config.js
│       ├── tailwind.config.ts
│       └── tsconfig.json
├── node_modules/
│   └── ...
├── packages/
│   └── ...
├── package.json
├── pnpm-lock.yaml
├── pnpm-workspace.yaml
└── turbo.json

AND finally I can confirm the instrumentation file is in there:

find . -name "instrumentation*" -type f 
./apps/hub/.next/standalone/apps/hub/instrumentation.ts
./apps/hub/instrumentation.ts

But for some reason it's not being compiled.

Everything when running locally works just fine and I can export telemetry with no issues.

CC @icyJoseph @timneutkens any comments more than welcome🙏

EDIT: I'm suspicious about this path: ./apps/hub/.next/standalone/apps/hub/instrumentation.ts

[scinscinscin]

Not sure if this is the same issue that i'm experiencing, but I cannot, for the life of me, get Vercel to call my src/instrumentation.server.ts file (I'm using page extensions with .server.ts). I'm not sure how Vercel builds and runs next.js in its platform, so im not sure if its related with this

[icyJoseph]

Does it work locally though? I don't remember details right now but I was trying to improve the custom page extensions documentation and found some unexpected behavior with some files - like the opengraph images. Had to postpone that work due to Next Conf but I intend to get back to it eventually.