misskey
misskey copied to clipboard
misskey-dev
refactor: migrate to Yarn Berry (v3.2.1)
What
Migrate to Yarn Berry (v3.2.1)
Resolve #5858 Supersede #8472
Why
As yarn v3 (yarn berry) has worked with Misskey for a while and provides better performance when compiling while still retaining backwards compatibility with yarn v1, I feel that upgrading to yarn berry would be a smart choice.
Additional info
I've been building Misskey with yarn v3 for two months now both on my server for production and on my development machine with absolutely no problems. This PR was made simply by running yarn with yarn 3.2.1 and following the yarn migration instructions.
Further steps
- [ ] Documentation on Misskey Hub would have to be updated
- [x] Update workflow (especially mocha and e2e)
- [x] Update the install script (maybe docker too?)
@acid-chicken I'd personally like your feedback on this PR first if you don't mind, as you've been the most involved with Yarn Berry discussion.
Also, since there are opinions like https://github.com/misskey-dev/misskey/issues/5858#issuecomment-611626477, it would be better if Corepack could be used as a boot loader for Yarn berry (instead of placing .yarn/releases/yarn-3.2.1.cjs directly).
Running corepack enable in the repo folder didn't change anything, is there a proper way to enable corepack?
corepack enable is related to environment settings, so it will not change anything in the repository (excluding docs).
Can you confirm that deleting .yarn/releases/yarn-3.2.1.cjs with "packageManager" specified in package.json after running it still works?
Can you confirm that deleting .yarn/releases/yarn-3.2.1.cjs with "packageManager" specified in package.json after running it still works?
@acid-chicken yep!
Seems like the labeler bot adds back removed labels every commit ^^"
Also, since Yarn berry has built-in monorepo management system called workspaces, so it would be great (and probably easy) to also improve the current situation, which is awkwardly realized by scripts/install-packages.js and so on.
Done!
I assume packages/*/yarn.lock will no longer be used after yarn install should consolidate the lock files in /yarn.lock.
Also seems like removing yarn-3.2.1.cjs broke the linting workflows...
corepack enable is related to environment settings, so, for now, we have run it manually after setup-node (actions/setup-node#482).
Seems like the current lint fails because of
https://github.com/misskey-dev/misskey/blob/ebc2566130c785924380d504c76b270198bdc2c2/.github/workflows/lint.yml#L21-L22
causing
So hopefully it'll fix itself after this is merged.
Should I add
packages/*/yarn.lockto.gitignore?
No. You have to run yarn install first, then they can safely deleted.
Also yarn --cwd ./packages/* ... are should be rewritten to yarn workspaces * ...
https://github.com/misskey-dev/misskey/blob/ebc2566130c785924380d504c76b270198bdc2c2/.github/workflows/lint.yml#L24
@acid-chicken before I commit, something like this?
jobs:
workspaces:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
submodules: true
- uses: actions/setup-node@v3
with:
node-version: 18.x
cache: 'yarn'
cache-dependency-path: |
yarn.lock
- run: corepack enable
- run: yarn install
- run: yarn workspaces * lint
Yes. More to say, it seems that cache-dependency-path is no longer needed.
@acid-chicken Hmm... yarn install still fails in the workflow
YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
Turns out yarn workspaces <client/backend> lint isn't actually a valid command :smile:
Lint now fails at the expected place (since Misskey still has a lot of lint errors) :D
node_modulesに依存しているコードを修正する必要があると思います misskeyのコードベースは小規模ではないのでworkspacesへの移行は大変だと思います
I think you need to fix the code that depends on node_modules...
misskey codebase isn't small, so migrating to workspaces can be a heavy task.
@ishowta If that much migrating needs to be done, would it be worth it to change to PnP?
すみません、yarn pnpを使用したことが無いので分かりません。しかし、移行はより難しくなると思います。例えば依存パッケージ内に存在するcssファイルをcssから参照することは出来るのでしょうか…
良し悪しは分かりませんが、hoistingを無効にして各ディレクトリにnode_modulesを配置するオプションがあります。 https://yarnpkg.com/configuration/yarnrc#nmHoistingLimits
workspacesの機能はyarn v1にも存在するので、berryへのアップグレードとは独立して考えたほうがいいと思います。
Sorry, I don't know, I have never used yarn pnp. But I think the transition will be more difficult. For example, is it possible to reference css files that exist in dependent packages...
I don't know if it is good or bad, but there is an option to disable hoisting and put node_modules in each directory. https://yarnpkg.com/configuration/yarnrc#nmHoistingLimits
The workspaces feature is also present in yarn v1, so I think it should be considered independent of the upgrade to berry.
I should add that it builds and runs fine in production on https://stop.voring.me but I'm not sure if I did everything right (plus, workflow is still broken)
すみません、yarn pnpを使用したことが無いので分かりません。しかし、移行はより難しくなると思います。例えば依存パッケージ内に存在するcssファイルをcssから参照することは出来るのでしょうか…
良し悪しは分かりませんが、hoistingを無効にして各ディレクトリにnode_modulesを配置するオプションがあります。 https://yarnpkg.com/configuration/yarnrc#nmHoistingLimits
workspacesの機能はyarn v1にも存在するので、berryへのアップグレードとは独立して考えたほうがいいと思います。
@ishowta https://github.com/misskey-dev/misskey/pull/8764#pullrequestreview-988673362 でも触れましたが、もちろん PnP mode を最初から有効にするのは移行コストが重いので、この PR では node_modules リンカによる移行で良いと思います（し、現状の差分ではそうなっています）。PnP を使用するかどうかと Yarn berry への移行を行うこと自体は別の話です。なお、node_modules パス周りで若干の修正が入っているのは、元々のモノリポ構成が手動で実現されていた点が経緯にあるものと思われます。
@acid-chicken あー私も同意見です。node_modulesがhoistingされている場合にパスを修正しなければいけないということを指して「コードを修正する必要があると思います。…workspacesへの移行は大変だと思います。」と言ったつもりでした。私が~英語を~文の意味を読み間違えて的はずれな返答をしてしまっていました。すみません。
もしyarn v2のlockfileフォーマットに対応していないパッケージがあれば修正する必要がありますが、それくらいだと思います。（テストが適切に行われていれば）
If there are packages that don't support the yarn v2 lockfile format, they will need to be modified, but I think that's about it. (As long as testing is done properly.)
Opening up again for review as build works fine, but workflows are still broken. Help would be greatly appreciated :pray: