build-image
build-image copied to clipboard
Published
20 hours ago •
netlify
netlify
Yarn 3 workspaces are not cached at all
Related: #612 #432 #319
We are using yarn workspaces with yarn's latest version (3.0.1) and yarn is not cached at all, it takes 1m 54s in best cases and 3m 17s in worst cases, including the cache restore phase.
On the contrary, on GitHub actions with the yarn cache action, the Fetch phase is completely skipped, and it just runs the linking phase. It takes ~20 seconds.
I pasted the logs at the end of the issue, the relevant part are these:
And there is this warning from yarn, it may be relevant:
Full Netlify build logs (excluding build phase)
4:18:09 PM: Build ready to start
4:18:11 PM: build-image version: c6001ed68662a13e5deb24abec2b46058c58248a
4:18:11 PM: build-image tag: v3.9.0
4:18:11 PM: buildbot version: d2795bf5a96b85e585e90944f15f15381ce164a7
4:18:11 PM: Fetching cached dependencies
4:18:11 PM: Starting to download cache of 802.8MB
4:18:15 PM: Finished downloading cache in 3.951667197s
4:18:15 PM: Starting to extract cache
4:18:31 PM: Finished extracting cache in 16.395963806s
4:18:32 PM: Finished fetching cache in 20.470693303s
4:18:32 PM: Starting to prepare the repo for build
4:18:32 PM: Preparing Git Reference pull/1028/head
4:18:35 PM: Parsing package.json dependencies
4:18:35 PM: Starting build script
4:18:35 PM: Installing dependencies
4:18:35 PM: Python version set to 2.7
4:18:36 PM: Started restoring cached node version
4:18:38 PM: Finished restoring cached node version
4:18:39 PM: v16.7.0 is already installed.
4:18:40 PM: Now using node v16.7.0 (npm v7.20.3)
4:18:40 PM: Started restoring cached build plugins
4:18:40 PM: Finished restoring cached build plugins
4:18:40 PM: Attempting ruby version 2.7.1, read from environment
4:18:40 PM: Started restoring cached ruby version
4:18:40 PM: Finished restoring cached ruby version
4:18:42 PM: Using ruby version 2.7.1
4:18:42 PM: Using PHP version 5.6
4:18:42 PM: Started restoring cached yarn cache
4:18:42 PM: Finished restoring cached yarn cache
4:18:44 PM: Found yarn version (3.0.1) that doesn't match expected (1.22.4)
4:18:44 PM: up to date, audited 1 package in 261ms
4:18:44 PM: found 0 vulnerabilities
4:18:44 PM: Installing yarn at version 1.22.4
4:18:44 PM: Installing Yarn!
4:18:44 PM: > Downloading tarball...
4:18:44 PM: [1/2]: https://yarnpkg.com/downloads/1.22.4/yarn-v1.22.4.tar.gz --> /tmp/yarn.tar.gz.uavJVbtnLD
4:18:44 PM: % Total % Received % Xferd Average Speed Time Time Time Current
4:18:44 PM: Dload Upload Total Spent Left Speed
4:18:45 PM: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
4:18:45 PM: 100 79 100 79 0 0 351 0 --:--:-- --:--:-- --:--:-- 352
4:18:45 PM: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
4:18:45 PM: 100 93 100 93 0 0 146 0 --:--:-- --:--:-- --:--:-- 451
4:18:45 PM: 100 625 100 625 0 0 684 0 --:--:-- --:--:-- --:--:-- 684
4:18:46 PM: 100 1215k 100 1215k 0 0 1006k 0 0:00:01 0:00:01 --:--:-- 1006k
4:18:46 PM: [2/2]: https://yarnpkg.com/downloads/1.22.4/yarn-v1.22.4.tar.gz.asc --> /tmp/yarn.tar.gz.uavJVbtnLD.asc
4:18:46 PM: 100 83 100 83 0 0 1853 0 --:--:-- --:--:-- --:--:-- 1853
4:18:46 PM: 100 97 100 97 0 0 1318 0 --:--:-- --:--:-- --:--:-- 1318
4:18:46 PM: 100 629 100 629 0 0 6041 0 --:--:-- --:--:-- --:--:-- 6041
4:18:46 PM: 100 1028 100 1028 0 0 8914 0 --:--:-- --:--:-- --:--:-- 8914
4:18:46 PM: > Verifying integrity...
4:18:46 PM: gpg: Signature made Mon 09 Mar 2020 03:52:13 PM UTC using RSA key ID 69475BAA
4:18:46 PM: gpg: Good signature from "Yarn Packaging <[email protected]>"
4:18:46 PM: gpg: WARNING: This key is not certified with a trusted signature!
4:18:46 PM: gpg: There is no indication that the signature belongs to the owner.
4:18:46 PM: Primary key fingerprint: 72EC F46A 56B4 AD39 C907 BBB7 1646 B01B 86E5 0310
4:18:46 PM: Subkey fingerprint: 6D98 490C 6F1A CDDD 448E 4595 4F77 6793 6947 5BAA
4:18:46 PM: > GPG signature looks good
4:18:46 PM: > Extracting to ~/.yarn...
4:18:46 PM: > Adding to $PATH...
4:18:46 PM: > Successfully installed Yarn 1.22.4! Please open another terminal where the `yarn` command will now be available.
4:18:46 PM: Yarn workspaces detected
4:18:46 PM: Started restoring workspace packages/editor node modules
4:18:46 PM: Finished restoring workspace packages/editor node modules
4:18:46 PM: Started restoring workspace root node modules
4:18:46 PM: Finished restoring workspace root node modules
4:18:47 PM: Installing NPM modules using Yarn version 3.0.1
4:18:48 PM: ➤ YN0050: The cache-folder option has been deprecated; use rc settings instead
4:18:49 PM: ➤ YN0000: ┌ Resolution step
4:18:49 PM: ➤ YN0002: │ @mdx-js/loader@npm:1.6.22 doesn't provide react (p12550), requested by @mdx-js/react
➤ YN0002: │ @react-aria/menu@npm:3.1.4 [6b3b6] doesn't provide react-dom (p7793e), requested by @react-aria/overlays
➤ YN0002: │ @react-aria/tooltip@npm:3.1.1 [6b3b6] doesn't provide react-dom (p25457), requested by @react-aria/overlays
➤ YN0002: │ @storybook/core@npm:6.1.21 [13e62] doesn't provide typescript (p4c2e3), requested by fork-ts-checker-webpack-plugin
➤ YN0002: │ @storybook/preset-create-react-app@npm:3.1.7 [7e6ab] doesn't provide react-refresh (pf9dee), requested by @pmmmwh/react-refresh-webpack-plugin
➤ YN0002: │ @storybook/preset-create-react-app@npm:3.1.7 [7e6ab] doesn't provide typescript (p0ad40), requested by react-docgen-typescript-plugin
➤ YN0002: │ @storybook/preset-create-react-app@npm:3.1.7 [7e6ab] doesn't provide webpack (p82525), requested by @pmmmwh/react-refresh-webpack-plugin
➤ YN0002: │ @storybook/react@npm:6.1.21 [7e6ab] doesn't provide typescript (p62333), requested by react-docgen-typescript-plugin
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @babel/core (p170a2), requested by babel-plugin-named-asset-import
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @babel/core (pf8a98), requested by @storybook/addon-essentials
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @babel/core (pc2008), requested by @storybook/preset-create-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @babel/core (p09571), requested by @storybook/react
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @babel/core (p1c95e), requested by babel-jest
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @testing-library/dom (p52ade), requested by @testing-library/user-event
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @typescript-eslint/eslint-plugin (pae93d), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @typescript-eslint/parser (p47980), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide babel-loader (pf2344), requested by @storybook/addon-docs
➤ YN0002: │ editor@workspace:packages/editor doesn't provide babel-loader (pd8908), requested by @storybook/addon-essentials
➤ YN0002: │ editor@workspace:packages/editor doesn't provide eslint-plugin-flowtype (pdeceb), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide eslint-plugin-import (p284ba), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide eslint-plugin-jsx-a11y (p2f943), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide eslint-plugin-react (p4b4fd), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide eslint-plugin-react-hooks (pb5a16), requested by eslint-config-react-app
➤ YN0060: │ editor@workspace:packages/editor provides react (pdd751) with version 17.0.1, which doesn't satisfy what @storybook/addon-actions and some of its descendants request
➤ YN0060: │ editor@workspace:packages/editor provides react (pa0ab2) with version 17.0.1, which doesn't satisfy what @storybook/addon-docs and some of its descendants request
➤ YN0060: │ editor@workspace:packages/editor provides react (p03409) with version 17.0.1, which doesn't satisfy what react-multi-email requests
➤ YN0060: │ editor@workspace:packages/editor provides react-dom (p60341) with version 17.0.1, which doesn't satisfy what @storybook/addon-actions and some of its descendants request
➤ YN0060: │ editor@workspace:packages/editor provides react-dom (pdc343) with version 17.0.1, which doesn't satisfy what @storybook/addon-docs and some of its descendants request
➤ YN0060: │ editor@workspace:packages/editor provides react-dom (pce5e4) with version 17.0.1, which doesn't satisfy what react-multi-email requests
➤ YN0002: │ editor@workspace:packages/editor doesn't provide webpack (pa7cad), requested by raw-loader
➤ YN0002: │ editor@workspace:packages/editor doesn't provide webpack (p68732), requested by esbuild-loader
➤ YN0002: │ editor@workspace:packages/editor doesn't provide webpack (p4ed41), requested by fast-sass-loader
➤ YN0002: │ editor@workspace:packages/editor doesn't provide webpack (p92b4b), requested by speed-measure-webpack-plugin
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pbc588), requested by @react-aria/i18n
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p8a0d0), requested by @react-aria/interactions
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p48a46), requested by @react-aria/link
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p109e9), requested by @react-aria/utils
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pc15d4), requested by @react-aria/focus
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p6b6be), requested by @react-aria/label
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p16962), requested by @react-aria/ssr
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p83932), requested by @react-aria/overlays
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p3ae77), requested by @react-aria/progress
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p80dff), requested by @react-aria/visually-hidden
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p3a742), requested by @react-aria/textfield
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pabf72), requested by @react-aria/menu
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pfc846), requested by @react-aria/breadcrumbs
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pcd1d8), requested by @react-aria/button
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p09bf0), requested by @react-aria/checkbox
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p51ee3), requested by @react-aria/dialog
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pb7f39), requested by @react-aria/listbox
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p4847b), requested by @react-aria/meter
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pc8bc1), requested by @react-aria/radio
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pc2954), requested by @react-aria/searchfield
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pfeef5), requested by @react-aria/select
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pede51), requested by @react-aria/separator
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p9db53), requested by @react-aria/slider
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pe751f), requested by @react-aria/switch
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p5cd5d), requested by @react-aria/tooltip
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react-dom (p337a3), requested by @react-aria/overlays
➤ YN0002: │ react-dev-utils@npm:11.0.4 doesn't provide typescript (p79ddf), requested by fork-ts-checker-webpack-plugin
➤ YN0002: │ react-dev-utils@npm:11.0.4 doesn't provide webpack (p2af19), requested by fork-ts-checker-webpack-plugin
➤ YN0002: │ react-hot-toast@npm:2.0.0 [7e6ab] doesn't provide csstype (p287b3), requested by goober
➤ YN0002: │ react-stately@npm:3.3.0 doesn't provide react (p85986), requested by @react-stately/data
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide @typescript-eslint/eslint-plugin (pdf9d2), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide @typescript-eslint/parser (pd2e4e), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide babel-eslint (peb234), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide cypress (pa68a9), requested by @testing-library/cypress
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide cypress (p85358), requested by cypress-localstorage-commands
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide eslint-plugin-flowtype (pfa493), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide eslint-plugin-import (pbae0d), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide eslint-plugin-jsx-a11y (pc5b66), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide eslint-plugin-react (p45cfd), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide eslint-plugin-react-hooks (pc0ad7), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide react (p5a3eb), requested by react-scripts
➤ YN0002: │ runtime@workspace:packages/runtime doesn't provide webpack (p462c7), requested by raw-loader
➤ YN0002: │ runtime@workspace:packages/runtime [7e6ab] doesn't provide webpack (pc8bdf), requested by raw-loader
➤ YN0002: │ spe@workspace:packages/spe doesn't provide three (p422b3), requested by postprocessing
➤ YN0002: │ spe@workspace:packages/spe doesn't provide webpack (pfa9bb), requested by file-loader
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
4:18:49 PM: ➤ YN0000: └ Completed in 0s 608ms
4:18:49 PM: ➤ YN0000: ┌ Fetch step
4:20:23 PM: ➤ YN0013: │ 8 packages were already cached, 2408 had to be fetched
4:20:23 PM: ➤ YN0000: └ Completed in 1m 34s
4:20:23 PM: ➤ YN0000: ┌ Link step
4:20:24 PM: ➤ YN0062: │ fsevents@patch:fsevents@npm%3A2.3.2#~builtin<compat/fsevents>::version=2.3.2&hash=1cc4b2 The platform linux is incompatible with this module, link skipped.
4:20:25 PM: ➤ YN0062: │ fsevents@patch:fsevents@npm%3A1.2.13#~builtin<compat/fsevents>::version=1.2.13&hash=1cc4b2 The platform linux is incompatible with this module, link skipped.
4:20:29 PM: ➤ YN0007: │ husky@npm:4.3.7 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ core-js@npm:3.9.1 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ electron@npm:8.5.5 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ esbuild@npm:0.12.19 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ ejs@npm:2.7.4 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ core-js-pure@npm:3.9.1 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ core-js@npm:2.6.12 must be built because it never has been before or the last one failed
4:20:30 PM: ➤ YN0000: │ husky@npm:4.3.7 STDOUT husky > Setting up git hooks
4:20:30 PM: ➤ YN0000: │ husky@npm:4.3.7 STDOUT CI detected, skipping Git hooks installation.
4:20:30 PM: ➤ YN0000: │ husky@npm:4.3.7 STDOUT husky > Done
4:20:31 PM: ➤ YN0000: └ Completed in 7s 690ms
4:20:31 PM: ➤ YN0000: Done with warnings in 1m 43s
4:20:32 PM: NPM modules installed using Yarn
4:20:33 PM: Started restoring cached go cache
4:20:33 PM: Finished restoring cached go cache
4:20:33 PM: go version go1.14.4 linux/amd64
4:20:33 PM: go version go1.14.4 linux/amd64
4:20:33 PM: Installing missing commands
4:20:33 PM: Verify run directory
4:20:34 PM:
4:20:34 PM: ────────────────────────────────────────────────────────────────
4:20:34 PM: Netlify Build
4:20:34 PM: ────────────────────────────────────────────────────────────────
4:20:34 PM:
4:20:34 PM: ❯ Version
4:20:34 PM: @netlify/build 18.5.0
4:20:34 PM:
4:20:34 PM: ❯ Flags
4:20:34 PM: baseRelDir: true
4:20:34 PM: buildId: 612651213fe4cc0008c22123
4:20:34 PM: deployId: 612651213fe4cc0008c22125
4:20:34 PM:
4:20:34 PM: ❯ Current directory
4:20:34 PM: /opt/build/repo
4:20:34 PM:
4:20:34 PM: ❯ Config file
4:20:34 PM: /opt/build/repo/netlify.toml
4:20:34 PM:
4:20:34 PM: ❯ Context
4:20:34 PM: deploy-preview
4:20:35 PM:
4:20:35 PM: ────────────────────────────────────────────────────────────────
4:20:35 PM: 1. Build command from Netlify app
4:20:35 PM: ────────────────────────────────────────────────────────────────
4:20:35 PM:
4:20:35 PM: $ yarn build
Thanks for raising this @marcofugaro. This is definitely related with #612 and #319 and our usage of the --cache-folder property 👍 however it isn't particular to your workspace settings I believe, just the yarn version used.
This issue is five months old today and there is still no solution for Yarn 3 in the public. I tried finding a workaround today but gave up: https://github.com/netlify/build/issues/1535#issuecomment-1021720396 HALP! 🙏
UPD: Workaround: https://github.com/netlify/build/issues/1535#issuecomment-1022988981
Hi, manage to create a workaround for yarn3 using a custom plugin, till #612
First in netlify.toml use NPM_FLAGS hack to disable default npm install, here we cannot use NETLIFY_USE_YARN becase of #779
[build.environment]
NODE_VERSION = "16"
NPM_FLAGS = "--version"
[[plugins]]
package = "/.netlify/netlify-plugin-yarn"
then create .netlify/netlify-plugin-yarn folder with files
- manifest.yml
- index.js
manifest.yml is just
name: netlify-plugin-yarn
inputs: []
and index.js is responsible for moving cache folder, setting YARN_CACHE_FOLDER and enabling corepack for packageManager in package.json
module.exports = {
onPreBuild: async ({ utils: { build, run } }) => {
const cacheDir = '.yarn_cache'
process.env['YARN_CACHE_FOLDER'] = `${process.env.HOME}/${cacheDir}`
process.env['NETLIFY_CACHE_DIR'] = `${process.env.NETLIFY_BUILD_BASE}/cache`
try {
await run.command(`mv ${process.env.NETLIFY_CACHE_DIR}/${cacheDir} ${process.env.YARN_CACHE_FOLDER}`)
} catch (error) {
console.log('No yarn cache found')
}
try {
await run.command('corepack enable')
} catch (error) {
return build.failBuild(error)
}
},
}
package.json
{
"private": true,
"engines": {
"node": "16.x",
"yarn": "3.2.1"
},
"packageManager": "[email protected]"
}
Edited ---
Extracted this code into package, https://github.com/piecyk/yarn-berry-netlify-plugin
