monorepo, when `pnpm install` in root, node_modules in packages are not created always

[valerii15298]

pnpm version: 6.30.1

Code to reproduce the issue:

Note: .npmrc config is: shared-workspace-lockfile=true

here is small reproduction: https://github.com/valerii15298/pnpm-repr.git

git clone https://github.com/valerii15298/pnpm-repr.git
cd pnpm-repr
pnpm install
pnpm clean:all

pnpm clean:all will delete pnpm-lock and node modules in packages

run: pnpm install - then just pnpm-lock file will be created run ls -r packages - to ensure nothing else is changed. And then again run pnpm install - node_modules in packages will be created. So we need two times run pnpm install to create node_modules in packages, but that is not all.

run pnpm clean:all - delete pnpm-lock and node_modules in packages pnpm install - just pnpm-lock.yaml will be created rm pnpm-lock.yaml pnpm install again - and again just pnpm-lock.yaml will be created rm pnpm-lock.yaml again to remove lock file

So command pnpm install seems to only create lock file in such case.

But now try to change packages/p1/package.json and add to dependencies: "react": "*" And now run pnpm install again - now node_modules will be created in packages/p1, but packages/p2 will be the same So pnpm install behaves differently(depending on existing root node-modules I suppose).

Expected behavior:

I would expect command to run always in the same way, so if it creates node_modules in packages then it must create it in all packages or no one(then we need run pnpm install again) and not to depend on root node_modules.

Why? It is confusing when a command behaves differently depending on something you don't know.

Actual behavior:

When lock file is not present, pnpm install adds node_modules to package only when package.json was modified. So for example when we added react to package.json(e. g.) made changes to package.json, then after pnpm install it added node_modeles and all dependencies(including wavesurfer.js) to only this package.

Additional information:

• node -v prints: v16.13.1
• Windows, macOS, or Linux?: Windows

Edit 1: also it causes problems when deploying to docker when using pnpm fetch with docker as described in the docs: https://pnpm.io/cli/fetch#usage-scenario The thing is: when using pnpm fetch it causes docker to cache. Then when pnpm install it does not create node_modules in all packages(just in some of them), this is how I met this problem: node_modules were not created in one package when deploying to docker. So I needed run pnpm install -r --offline two times in Dockerfile.

Edit 2: Luckily for problem with docker found a solution to use the option --frozen-lockfile. After that node_modules were installed in all packages.

[samzmann]

I'm having the same issue. Have a pnpm-workspace.yaml that looks like this

packages:
  - 'packages/**'
  - 'apps/**'

Install seems to go through:

Progress: resolved 864, reused 205, downloaded 660, added 0

But node_modules dir is nowhere to be found...

[DylanCulfogienis]

Also reporting this issue, running pnpm install after rebasing will sometimes cause node_modules to not be updated with new additions.

Edit: I specifically encounter this when the following .npmrc setting is set:

auto-install-peers=true

If this is set, AND there are peer dependency resolution failures, it will result in a silent failure.

[fishsticks89]

I get this issue on linux even on 8.3.1 (not on windows)

[artemis-prime]

Still seeing this in 8.12.1... what the heck? How is this not being given attention??

[weyert]

You can always try to fix it yourself and contribute a PR

[hrutik7]

any solution on this ?

[reza-ebrahimi]

Workaround:

$ # in my case next, replace it with one of other packages you are using. 
$ # it will force to create node_modules directory
$ pnpm add next 
$ pnpm install # this command is not necessary since the add command above installs everything.

But this is actually a bug and needs to be fixed.

[valerii15298]

@reza-ebrahimi exactly! I use the same workaround. Basically just adding dummy package and then remove it like this:

pnpm add smallest
pnpm remove smallest

[bitcooker]

pnpm version: 6.30.1

Code to reproduce the issue:

Note: .npmrc config is: shared-workspace-lockfile=true

here is small reproduction: https://github.com/valerii15298/pnpm-repr.git

git clone https://github.com/valerii15298/pnpm-repr.git
cd pnpm-repr
pnpm install
pnpm clean:all

pnpm clean:all will delete pnpm-lock and node modules in packages

run: pnpm install - then just pnpm-lock file will be created run ls -r packages - to ensure nothing else is changed. And then again run pnpm install - node_modules in packages will be created. So we need two times run pnpm install to create node_modules in packages, but that is not all.

run pnpm clean:all - delete pnpm-lock and node_modules in packages pnpm install - just pnpm-lock.yaml will be created rm pnpm-lock.yaml pnpm install again - and again just pnpm-lock.yaml will be created rm pnpm-lock.yaml again to remove lock file

So command pnpm install seems to only create lock file in such case.

But now try to change packages/p1/package.json and add to dependencies: "react": "*" And now run pnpm install again - now node_modules will be created in packages/p1, but packages/p2 will be the same So pnpm install behaves differently(depending on existing root node-modules I suppose).

Expected behavior:

I would expect command to run always in the same way, so if it creates node_modules in packages then it must create it in all packages or no one(then we need run pnpm install again) and not to depend on root node_modules.

Why? It is confusing when a command behaves differently depending on something you don't know.

Actual behavior:

When lock file is not present, pnpm install adds node_modules to package only when package.json was modified. So for example when we added react to package.json(e. g.) made changes to package.json, then after pnpm install it added node_modeles and all dependencies(including wavesurfer.js) to only this package.

Additional information:

• node -v prints: v16.13.1
• Windows, macOS, or Linux?: Windows

Edit 1: also it causes problems when deploying to docker when using pnpm fetch with docker as described in the docs: https://pnpm.io/cli/fetch#usage-scenario The thing is: when using pnpm fetch it causes docker to cache. Then when pnpm install it does not create node_modules in all packages(just in some of them), this is how I met this problem: node_modules were not created in one package when deploying to docker. So I needed run pnpm install -r --offline two times in Dockerfile.

Edit 2: Luckily for problem with docker found a solution to use the option --frozen-lockfile. After that node_modules were installed in all packages.

I added shared-workspace-lockfile=true in the .npmrc file and it works.

[insanity54]

I've been bitten by this issue. I have a monorepo with 4 packages and pnpm install is not creating any node_modules directories. I'm hoping to use the pnpm add workaround suggested at https://github.com/pnpm/pnpm/issues/4321#issuecomment-1978832698 and https://github.com/pnpm/pnpm/issues/4321#issuecomment-1978832698, but I'm not sure how to use it. where in the dockerfile is the pnpm add supposed to go? Before or after pnpm install? Somewhere else?

I tried the pnpm add workaround at a few places. Didn't seem to have any effect. I still don't have any node_modules directories anywhere in the docker image.

I tried putting pnpm add in a few places in my dockerfile and I realized that I was adding the package to the root of the workspace, which is not what I want. It generates a pnpm ERR_PNPM_ADDING_TO_ROOT error which is understandable.

So I thought to change the docker WORKDIR to the directory of one of the workspace packages, and I still got the ERR_PNPM_ADDING_TO_ROOT error. That seems very wrong, because /usr/src/app is where my pnpm-workspace.yaml file is at. /usr/src/app is the workspace root, not /usr/src/app/packages/bot. Could that be part of this bug, where pnpm incorrectly acts as if a package directory is the root workspace directory?

I don't know how to debug this, because I can't find the code that handles pnpm install. I tried searching for the string, 'ERR_PNPM_ADDING_TO_ROOT' but the only search results I got were from a test file.

Here is my dockerfile.

FROM node:20.15 as base
ENV PNPM_HOME="/pnpm"
ENV PATH="$PNPM_HOME:$PATH"
WORKDIR /usr/src/app
RUN corepack enable && corepack prepare [email protected] --activate


FROM base as install
ENV NODE_ENV development
RUN mkdir -p /usr/src/app/packages/bot && mkdir /usr/src/app/packages/temporal-workflows && mkdir -p /prod/bot
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml .npmrc .
COPY ./packages/bot/pnpm-lock.yaml ./packages/bot/
COPY ./packages/temporal-workflows/pnpm-lock.yaml ./packages/temporal-workflows/
COPY ./packages/types/pnpm-lock.yaml ./packages/types/
WORKDIR /usr/src/app/packages/bot
RUN ls -lash .
RUN pwd
## => /usr/src/app/packages/bot
## this is where the weirdness happens.
RUN pnpm add discord.js
## => ERR_PNPM_ADDING_TO_ROOT
## pnpm thinks we're in the workspace root!?!?
RUN pnpm remove discord.js
WORKDIR /usr/src/app
RUN pnpm fetch
RUN --mount=type=cache,id=pnpm,target=/pnpm/store pnpm install --frozen-lockfile
COPY ./packages/bot/ ./packages/bot/
COPY ./packages/temporal-workflows/ ./packages/temporal-workflows/
COPY ./packages/types/ ./packages/types/
RUN echo "@@@@@@@@@@@@@@@@@@@@ show me the node_modules! (there are none)"
RUN ls -lash ./
RUN ls -lash ./packages
RUN ls -lash ./packages/bot
ENTRYPOINT ["pnpm"]

FROM install AS build
RUN pnpm -r build
RUN pnpm deploy --filter=bot --prod /prod/bot
RUN pnpm deploy --filter=bot /prod/bot-dev

FROM base as bot
WORKDIR /app
ENV NODE_ENV production
COPY --from=build /prod/bot .
RUN ls -la .
CMD ["start"]

FROM install as dev
WORKDIR /app
RUN ls -la .
COPY --from=build /prod/bot-dev .
CMD ["run", "dev"]

Edit 1 Here's a reproduction repository https://github.com/insanity54/pnpm-4321