midway
midway copied to clipboard
Published
20 hours ago •
midwayjs
midwayjs
[QUESION]有没有什么办法可以让pnpm prune后项目跑起来
背景：
RUN pnpm i --shamefully-hoist --registry=https://registry.npmmirror.com
RUN pnpm run build
RUN pnpm prune --production
运行的时候报错，说没找到
egg-session
egg-onerror
...
然后我再次安装egg，没有效果，需要一个一个安装这些子依赖，很奇怪不知道pnpm是怎么搞的
- 如果使用pnpm i -P正常
- 如果使用yarn npm这些也正常
所以就是如果pnpm prune后 本该引用egg的依赖就没了
用pnpm的原因
- 如果用npm的话 安装依赖这个环节就花费了1分多钟，总体时间都到3分钟外了
- 如果使用pnpm的话，安装依赖只要20几秒，总体时间可以压缩在1分钟左右
- 效益非常高
需求
- 如何解决这种情况
- 有没有其他办法再提高构建速度，现在一个项目3分多钟太慢了，
三分钟应该是包含了依赖安装吧。
没错
我现在
FROM node:12 AS build
WORKDIR /app
COPY . .
RUN npm i --registry=https://registry.npmmirror.com
RUN npm run build
RUN npm prune --production
# 生成生产镜像
FROM node:12-alpine
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/bootstrap.js ./
COPY --from=build /app/package.json ./
COPY --from=build /app/node_modules ./node_modules
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN apk add --no-cache tzdata
ENV TZ="Asia/Shanghai"
# RUN npm install --production --registry=https://registry.npm.taobao.org
EXPOSE 7001
CMD ["npm", "run", "start"]
感觉优化不动了，就卡在依赖构建这速度太慢，然后用yarn的速度居然和npm差不多，所以用了pnpm
纯midwayDemo 如果用npm/yarn = 最快也要130s 如果用pnpm=最快60s 这个还相对合理 毕竟其他语言构建不需要这么久
然后还有尝试的就是
- 在搞个中间构建(jenkins)，把依赖缓存在构建环境里，dockerfile只做复制文件。这样速度是最快的，但是多了中间一层，项目配置变得繁琐，适用性也不高
- 再搞个docker环境，里面就是纯依赖，和上面一个其实差不多，但更繁琐
- 不运行prune，牺牲文件大小和上传速度。原来113m，现在是330m
- prune之后在运行一次pnpm i -P，这个效果还行。就看起来又傻有不合理
[+] Building 69.3s (22/22) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 861B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/node:12-alpine 2.4s
=> [internal] load metadata for docker.io/library/node:12 2.4s
=> [build 1/8] FROM docker.io/library/node:12@sha256:cb615135757e6e2eed4fad6d80f07b13d9f35072700c01da6de26fde5c9e8632 0.0s
=> [internal] load build context 4.1s
=> => transferring context: 3.44MB 4.0s
=> [stage-1 1/8] FROM docker.io/library/node:12-alpine@sha256:0eca266c5fe38ba93aebac00e45c9ac1bb7328b0702a6dc10e1a6ea543d49301 0.0s
=> CACHED [build 2/8] WORKDIR /app 0.0s
=> [build 3/8] COPY . . 5.9s
=> [build 4/8] RUN npm i pnpm -g --registry=https://registry.npmmirror.com 3.5s
=> [build 5/8] RUN pnpm i --prefer-offline -frozen-lockfile --shamefully-hoist --registry=https://registry.npmmirror.com 22.6s
=> [build 6/8] RUN npm run build 11.9s
=> [build 7/8] RUN pnpm prune --production 8.0s
=> [build 8/8] RUN pnpm i --prefer-offline -frozen-lockfile --shamefully-hoist --registry=https://registry.npmmirror.com -P 6.7s
=> CACHED [stage-1 2/8] WORKDIR /app 0.0s
=> CACHED [stage-1 3/8] COPY --from=build /app/dist ./dist 0.0s
=> CACHED [stage-1 4/8] COPY --from=build /app/bootstrap.js ./ 0.0s
=> CACHED [stage-1 5/8] COPY --from=build /app/package.json ./ 0.0s
=> [stage-1 6/8] COPY --from=build /app/node_modules ./node_modules 0.9s
=> [stage-1 7/8] RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories 0.3s
=> [stage-1 8/8] RUN apk add --no-cache tzdata 1.4s
=> exporting to image 0.9s
=> => exporting layers 0.9s
=> => writing image sha256:676469aed33ab67511569a062d8ffbabf5d049cb621462fddda5b366a65fc6be 0.0s
=> => naming to docker.io/library/midway-04 0.0s
你不是想要小一些吗？挺爷不是搞了ncc bundle的那个，加上业务貌似只有几M，然后如果带上node则40来M。
牛逼 感觉这条路也不错 就是打包成单文件就像其他语言构建成二进制一样
实际还是要看业务引的包，现在只是纯框架+示例的大小。
没事 大小还能接受 现在就是觉得构建时间太久了 在安装依赖这个环节 我们其他go项目构建加上传也才15s，node要花3分多钟，所以希望能加快点构建速度，这样修线上bug也快点
想知道大家有没有优化的方法
极致优化可以在docker构建之外进行。我的方法是在 gitlab-ci 中构建
- ci 中缓存 node_modules 目录
- Dockerfile 构建脚本根据配置参数分批次对 node_modules 的子目录进行拷贝，从而最大限度复用历史镜像层
Docker 镜像整体也就 70M 左右，每次新构建的 docker 镜像基本上就业务代码那层以及一两依赖层会有变动，大概四五MB。极大节省了 docker 存储容量。
极致优化可以在docker构建之外进行。我的方法是在 gitlab-ci 中构建
- ci 中缓存 node_modules 目录
- Dockerfile 构建脚本根据配置参数分批次对 node_modules 的子目录进行拷贝，从而最大限度复用历史镜像层
Docker 镜像整体也就 70M 左右，每次新构建的 docker 镜像基本上就业务代码那层以及一两依赖层会有变动，大概四五MB。极大节省了 docker 存储容量。
这个方案是在对第三方依赖变动很少的场景下效果比较明显吗？
极致优化可以在docker构建之外进行。我的方法是在 gitlab-ci 中构建
- ci 中缓存 node_modules 目录
- Dockerfile 构建脚本根据配置参数分批次对 node_modules 的子目录进行拷贝，从而最大限度复用历史镜像层
Docker 镜像整体也就 70M 左右，每次新构建的 docker 镜像基本上就业务代码那层以及一两依赖层会有变动，大概四五MB。极大节省了 docker 存储容量。
这个方案是在对第三方依赖变动很少的场景下效果比较明显吗？
是的，效果很明显。 以下优化供参考，且仅适用于 midway.js （cjs）
export const mods = [
[
"bintrees",
"braces",
"bufrw",
"busboy",
"crc",
"esm",
"esprima",
"expand-brackets",
"ioredis",
"jsonwebtoken",
"long",
"moment",
"moment-timezone",
"psl",
"pg",
"pg-native",
"pg-pool",
"pg-protocol",
"reflect-metadata",
"request",
"snapdragon",
"sshpk",
"thriftrw",
"through2",
"tweetnacl",
], // 0
[
"@koa",
"@opentelemetry",
"koa-session",
"ast-types",
"common-bin",
"get-uri",
"jaeger-client",
"json-schema",
"logform",
"node-fetch",
"opentracing",
"qs",
"source-map-support",
"uri-js",
"uuid",
"winston",
"winston-transport",
], // 1
[
"ajv",
"joi",
"knex",
"lodash",
"rxjs",
"semver",
], // 2
[
"@midwayjs",
"@mwcp",
"@waiting",
"kmore",
], // 3
[
"koa",
"iconv-lite",
"ready-callback",
"wt",
], // 4
[
"are-we-there-yet",
"mysql2",
"needle",
"node-pre-gyp",
"sqlite3",
"xmlbuilder",
], // 5
]
上面把依赖划分组，构建docker 的时候分别拷贝： Dockerfile
WORKDIR /app
RUN mkdir -p /app/run \
&& chown -R node /app
COPY --chown=node node_modules-ext-0 ./node_modules
COPY --chown=node node_modules-ext-1 ./node_modules
COPY --chown=node node_modules-ext-2 ./node_modules
COPY --chown=node node_modules-ext-3 ./node_modules
COPY --chown=node node_modules-ext-4 ./node_modules
COPY --chown=node node_modules-ext-5 ./node_modules
COPY --chown=node node_modules ./node_modules
COPY --chown=node node_modules-ext-99 ./node_modules
COPY --chown=node dist ./dist
COPY --chown=node package.json README.md tsconfig.json bootstrap.js ./
频繁构建时基本就只有业务代码（层）才变动。
另外，还可以清理下 node_module 里面无关的文件，比如（供参考）：
clean-pkg-files.sh
#!/bin/bash
# Should called by tar.sh and build-images.sh
# Should under the folder <project>/packages/<some-pkg>
set -e
find . -mindepth 3 -type d -iname "docs" -print0 | xargs -P0 -0II rm -rf I
find . -type d -iname "@types" -print0 | xargs -P0 -0II rm -rf I
find . -type d -iname ".github" -print0 | xargs -P0 -0II rm -rf I
find . -type d -iname "benchmarks" -print0 | xargs -P0 -0II rm -rf I
find . -type d -iname "example" -print0 | xargs -P0 -0II rm -rf I
find . -type d -iname "man1" -print0 | xargs -P0 -0II rm -rf I
find . -type d -iname "spec" -print0 | xargs -P0 -0II rm -rf I
find . -type d -iname "test" -print0 | xargs -P0 -0II rm -rf I
find . -type d -iname "tests" -print0 | xargs -P0 -0II rm -rf I
# unlink for image build cache
find . -type f -iname "package-lock.json" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".package-lock.json" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".bak" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".coveralls.yml" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".DS_Store" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".dockerignore" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".editorconfig" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".eslint*" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".gitattributes" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".github" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".istanbul.yml" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".jshintrc" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".jscs.json" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".jscsrc" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".name" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".npmignore" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".nycrc" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".prettierrc*" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".travis.yml" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname ".tslint*" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "appveyor.yml" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "docker-compose.yml" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "Dockerfile" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "donate.png" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "example.png" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "flake.lock" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "float.patch" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "CHANGELOG*" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "CHANGELOG.rst" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "Makefile" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "test.js" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "yarn.lock" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "*.markdown" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "*.md" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "*.swp" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "*.html" -not -path "./koa-onerror/templates/*" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "*.d.ts" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "*.d.ts.map" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "*.d.cts" -print0 | xargs -P0 -0II rm -f I
find . -type f -iname "LICENCE*" -print0 | xargs -P0 -0II gzip I
find . -type f -iname "LICENSE*" -print0 | xargs -P0 -0II gzip I
cd ../
clean-extra-files.sh:
#!/bin/bash
# Should called by util/prepare-pkg.sh
# Should under the folder <project>/packages/<some-pkg>/node_modules
# remove files under subfolder
echo -e ">>> Purging extra files"
find $(find $(find . -type d -iname "@waiting") -mindepth 2 -maxdepth 2 -type d -name "dist") \
-maxdepth 1 -type f -iname "*.esm.*" -print0 | xargs -0II rm -f I
find $(find $(find . -type d -iname "@waiting") -mindepth 2 -maxdepth 2 -type d -name "dist") \
-maxdepth 1 -type f -iname "*.umd.*" -print0 | xargs -0II rm -f I
find $(find . -type d -iname "ajv") -maxdepth 1 -type d -iname "dist" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "ajv") -maxdepth 1 -type d -iname "scripts" -print0 | xargs -0II rm -rf I
# find $(find . -type d -iname "ali-oss") -maxdepth 1 -type d -iname "dist" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "joi") -maxdepth 1 -type d -iname "dist" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "json5") -maxdepth 1 -type d -iname "dist" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "moment") -maxdepth 1 -type d -iname "dist" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "moment") -maxdepth 1 -type d -iname "src" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "moment") -maxdepth 1 -type d -iname "min" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "moment-timezone") -maxdepth 1 -type d -iname "builds" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "nunjucks") -maxdepth 1 -type d -iname "browser" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "needle") -maxdepth 1 -type d -iname "examples" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "opentracing") -maxdepth 1 -type d -iname "src" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 1 -type d -iname "src" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 1 -type d -iname "_esm5" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 1 -type d -iname "_esm2015" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 1 -type d -iname "add" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 1 -type d -iname "migrations" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 1 -type d -iname "observable" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 1 -type d -iname "operator" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 1 -type d -iname "testing" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 2 -type d -iname "bundles" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 2 -type d -iname "esm5" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 2 -type d -iname "esm" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "rxjs") -maxdepth 2 -type d -iname "types" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "sqlite3") -maxdepth 1 -type d -iname "src" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "sqlite3") -maxdepth 1 -type d -iname "deps" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "sqlite3") -maxdepth 1 -type d -iname "tools" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "sqlite3") -maxdepth 1 -type d -iname "build-tmp-*" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "sqlite3") -maxdepth 1 -type d -iname "build-tmp-napi-v6" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "source-map") -maxdepth 1 -type d -iname "dist" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "winston") -maxdepth 1 -type d -iname "dist" -print0 | xargs -0II rm -rf I
# opentelemetry
find $(find . -type d -iname "@opentelemetry") -maxdepth 3 -type d -iname "esm" -print0 | xargs -0II rm -rf I
find $(find . -type d -iname "@opentelemetry") -maxdepth 3 -type d -iname "esnext" -print0 | xargs -0II rm -rf I
大致可以缩减 node_module 目录一半空间。
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}