turbo
turbo copied to clipboard
vercel
cannot copy to non-directory DOCKER MACOS
What version of Turborepo are you using?
1.4.7
What package manager are you using / does the bug impact?
pnpm
What operating system are you using?
Mac
Describe the Bug
On Mac, I tested on windows and this run perfectly, with this repo: https://github.com/yovanoc/turbo-test
if we run pnpm build:docker this give:
Expected Behavior
Should work on Mac too
To Reproduce
Just clone the repo and build with docker
Can you update docker? When I run this locally I error out of docker immediately. This isn't likely a turbo-related issue. I'm closing it for now, but if you can get me a reproduction I'll be happy to reopen it!
~/repos/turbo-test (main)$ pnpm build:docker
> @ build:docker ~/repos/turbo-test
> COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -f docker-compose.yml build --parallel
ERROR: 'network_mode' and 'networks' cannot be combined
ELIFECYCLE Command failed with exit code 1.
@nathanhammond really sorry just push it again was just some tests :/ You can try again
@tknickman I'm tagging you in on this one; can reproduce, but seems quite strange to me.
Wondering if it’s related to symlinks; what if you add node_modules to the .dockerignore file to avoid its including it as part of the build context? Especially when you say it works on Windows I think it could be symlinks related
Adding node_modules to .dockerignore doesn't solve this issue but yeah I think about symlinks related problem too
Yeah, not sure, if it also needs to include the Pnpm store directory or nested node_modules. You can also try if it works without Buildkit (DOCKER_BUILDKIT=0)
Yeah I also tested that the other time, and I don't know for others but me I have that:
I am having this issue as well. Also on a Mac (M1). Was not yet able to test on a Windows machine but will do. This is my (very similar) Dockerfile:
FROM node:alpine AS base
RUN apk update
# Globally install `turbo`
RUN yarn global add turbo
# Prune the workspace for the `tester` app
FROM base as pruner
WORKDIR /app
COPY . .
RUN turbo prune --scope=@wecoma-lite-tester/tester --docker
FROM base AS builder
WORKDIR /app
COPY .gitignore .gitignore
COPY --from=pruner /app/out/json/ .
COPY --from=pruner /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
COPY --from=pruner /app/out/pnpm-workspace.yaml ./pnpm-workspace.yaml
COPY .npmrc .
# Install the deps needed to build the target
RUN corepack enable
RUN pnpm install
# Copy source code of pruned subworkspace and build
COPY --from=pruner /app/out/full/ .
COPY turbo.json turbo.json
RUN pnpm turbo run build --scope=@wecoma-lite-tester/tester...
And the error I get:
I already tested all mentioned fix ideas but nothing worked for me.
I got the same issue: turbo: 1.5.5 docker desktop: v4.12.0 pnpm @7.5.0
somehow either pnpm install works or the copy of the full folder. but not both!
# ------------------------------------------------------------------
# Stage 0
# ------------------------------------------------------------------
FROM node:16-alpine AS base-image
RUN apk update &&\
apk add --no-cache libc6-compat &&\
apk --no-cache add curl &&\
curl -f https://get.pnpm.io/v6.16.js | node - add --global [email protected]
FROM base-image AS builder
WORKDIR /app
RUN npm install turbo -g
COPY . .
RUN pnpm turbo prune --scope=@app/myapp-client --docker
# Add lockfile and package.json's of isolated subworkspace
FROM base-image AS installer
WORKDIR /app
# First install dependencies (as they change less often)
COPY .gitignore .gitignore
COPY .npmrc .npmrc
COPY --from=builder /app/out/json/ ./
COPY --from=builder /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
COPY --from=builder /app/out/pnpm-workspace.yaml ./pnpm-workspace.yaml
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< either this works
# RUN pnpm install
# Build the project
COPY turbo.json turbo.json
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< or this works
COPY --from=builder /app/out/full/ .
I had this same issue & noticed that after running the prune command (even outside of Docker), each package in out/full had a node_modules with the expected file structure, but with no actual source code. I think everything that pnpm symlinked is just missing in the full pruned output, but the fact that the node_modules directory structure is there causes an issue? I was able to work around this by removing all node_modules from out/full -- this is my full working Dockerfile:
FROM node:16-alpine AS builder
RUN apk update
# Set working directory
WORKDIR /app
RUN corepack enable
COPY . .
RUN pnpm --package=turbo dlx turbo prune --scope=web --docker
# remove all empty node_modules folder structure
RUN rm -rf /app/out/full/*/*/node_modules
# Add lockfile and package.json's of isolated subworkspace
FROM node:16-alpine AS installer
RUN apk update
RUN corepack enable
WORKDIR /app
# First install the dependencies (as they change less often)
COPY .gitignore .gitignore
COPY --from=builder /app/out/json/ .
COPY --from=builder /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
RUN pnpm install
# Build the project
COPY turbo.json turbo.json
COPY --from=builder /app/out/full/ .
RUN pnpm turbo run build --filter=web...
FROM node:16-alpine AS runner
WORKDIR /app
ENV NODE_ENV production
ENV NEXT_TELEMETRY_DISABLED 1
# Don't run production as root
RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs
USER nextjs
# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=installer --chown=nextjs:nodejs /app/apps/web/.next/standalone ./
COPY --from=installer --chown=nextjs:nodejs /app/apps/web/.next/static ./.next/static
CMD node apps/web/server.js
I'm not actually sure what the intention is with the prune command though -- ie:, why does the with-docker example copy over just the pruned package jsons and lockfile before running pnpm install, and THEN copy over the full source code? Wouldn't the pruned lockfile & package.json just be installing everything already in out/full anyway? Why not just copy over out/full and then run install (I guess it's to do with Docker layer cacheing where the source code changes more frequently than the dependencies)?
Update to add full source code: https://github.com/redbadger/monorepo-examples/tree/turborepo-monobuild
I had this issue too with just node_modules, but **/node_modules will ignore all node_modules.
Had the same issue. It was fixed by adding **/node_modules to .dockerignore, like @John-Hejzlar said.
If docker can't copy 'out' dir of turbo during build and if you use M1 you can solve with this environment for docker
DOCKER_DEFAULT_PLATFORM=linux/amd64
Try add
**/node_modulesto.dockerignore
Works great. Just watch out, in my case Prettier was changing ** to \*\* and I was not sure if it would be a thing... it is ;) (use .prettierignore and list **/.dockerignore)
Just reproduced this with the with-docker example when using pnpm. It works great when using yarn. Adding the details if it helps to debug.
What version of Turborepo are you using?
1.6.3
What package manager are you using / does the bug impact?
pnpm
What operating system are you using?
Mac M1
To Reproduce
npx degit vercel/turbo/examples/with-docker with-docker\ncd with-docker
Update Dockerfile to use pnpm instead of yarn:
# This Dockerfile is copy-pasted into our main docs at /docs/handbook/deploying-with-docker.
# Make sure you update both files!
FROM node:alpine AS builder
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
RUN apk update
# Set working directory
WORKDIR /app
RUN yarn global add turbo pnpm
COPY . .
RUN turbo prune --scope=web --docker
# Add lockfile and package.json's of isolated subworkspace
FROM node:alpine AS installer
RUN apk add --no-cache libc6-compat
RUN apk update
WORKDIR /app
# First install the dependencies (as they change less often)
RUN yarn global add pnpm
COPY .gitignore .gitignore
COPY --from=builder /app/out/json/ .
COPY --from=builder /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
RUN pnpm install
# Build the project
COPY --from=builder /app/out/full/ .
COPY turbo.json turbo.json
RUN pnpm turbo run build --filter=web...
FROM node:alpine AS runner
WORKDIR /app
# Don't run production as root
RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs
USER nextjs
COPY --from=installer /app/apps/web/next.config.js .
COPY --from=installer /app/apps/web/package.json .
# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=installer --chown=nextjs:nodejs /app/apps/web/.next/standalone ./
COPY --from=installer --chown=nextjs:nodejs /app/apps/web/.next/static ./apps/web/.next/static
CMD node apps/web/server.js
Build image
rm yarn.lock
pnpm install
docker build -f apps/web/Dockerfile .
I had this issue too with just
node_modules, but**/node_moduleswill ignore all node_modules.
same issue on M1 and this work
solved by updating node_modules to **/node_modules in .dockerignore 😄
but, got the next lined up error: failed to compute cache key: "/app/apps/web/.next/standalone" not found: not found
@xencodes I ran into the same thing, I think the example build assumes you're using standalone output https://nextjs.org/docs/advanced-features/output-file-tracing#automatically-copying-traced-files
Yeah, not sure, if it also needs to include the Pnpm store directory or nested node_modules. You can also try if it works without Buildkit (DOCKER_BUILDKIT=0)
It works with DOCKER_BUILDKIT=0 docker compose build
https://github.com/vercel/turbo/discussions/3346#discussioncomment-5703415
Yeah it's the symlinks
(must include root node_modules - talking about docker-compose dev here)
Looks like we have some answers above that solve these Docker configuration issues. Will close!
