Switch to Yarn v3

[benpsnyder]

I have upgraded to Yarn v3 with node-linker across many projects and have found it to be quite stable.

I also sorted package.json using yarn dlx sort-package-json

[retlehs]

thanks for the PR! i think that this issue on the bud repo still needs to be resolved:

https://github.com/roots/bud/issues/387

[benpsnyder]

thanks for the PR! i think that this issue on the bud repo still needs to be resolved:

roots/bud#387

I've applied a workaround that should work fine until that issue is solved.

[benpsnyder]

can you please revert the changes made with sort-package-json? bud configuration was at the top intentionally for ease of use

Moved it back to the top for you

[retlehs]

thanks! fwiw, sage likely won't be switching to yarn v3 unless bud does

as far as i understand, there's been a ton of pushback in the dev community about yarn v3. this decision is in the hands of our budtenders, @kellymears and @QWp6t

[benpsnyder]

Totally understand. Here's my view for @kellymears and @QWp6t -- Yarn v3 with nodeLinker: node-modules and WITHOUT zero-installs

ref: .gitignore

.pnp.*
.yarn/*
!.yarn/patches
!.yarn/plugins
!.yarn/releases
!.yarn/sdks
!.yarn/versions

... is just Yarn v1 but WAY FREAKING FASTER 👯

instead of using yarn install --frozen-lockfile on Yarn v1 I just now use yarn install --immutable on Yarn v3. That is literally all the difference.

If bud cares about speed and developer experience, I think it will adopt Yarn v3 over Yarn v1 at least without zero-installs and with node-linker. Makes total sense.

[knowler]

I think we should use NPM instead of Yarn, but that’s just my two cents (it’s just one less thing for people to install). I haven’t actually looked into Yarn 3, only Yarn 2 which I didn’t adopt for a number of reasons. I’ll look into Yarn 3 though.

[kellymears]

I approve of this.

We use yarn v3 with node-linker in the roots/bud monorepo. I think it's a good tool and I agree it's a better overall tool than yarn v1 at this point. I think both are way friendlier than npm in terms of overall dev experience (especially when it comes to the cli). Also, pinch me, who doesn't love a good underdog story (seeing how npm is now a microsoft product!)

A year ago? No way I would have approved of this PR. We almost switched to npm. In fact, I think we would have if @qwp6t didn't have such a cool head and steady draw. Today, I am obviously very glad that we did not switch.

It is very likely that roots/bud will provide an (optional) yarn v3 plugin in a future release. I don't think we're fully pnp compatible, but that is likely the next big thing we take on (after we've fully transitioned to using esm). I think pnp is an excellent idea but replacing all of our filesystem operations with pnp resolver calls is going to be a proper task.

Things I appreciate about yarn v3:

• The flexibility afforded by the hackable dependency resolution strategy. pnp as a default resolver is an excellent choice and first party support of node-linker is deeply appreciated. I agree that node-linker is the correct strategy for sage.
• The yarn patch functionality is incredible. Instead of waiting for an upstream to accept a PR to fix a critical error you can patch it for your project. It is easy to do this and the cli dx is quite good.
• Yarn v3 has an API. This is huge. And that API supports plugins -- even better. All of the bud monorepo's internal tooling is handled with a yarn v3 plugin. I think there is room to explore this in roots/sage for project maintenance tasks that fall outside of the purview of bud (which is, ultimately, a build tool).
• Yarn v3's lockfile is valid yml. npm is json. Even worse, yarn v1's lockfile is some sort of monstrous almost yml. I've really come to appreciate having a readable lockfile...
• The yarn version that lives in the project is separate from the yarn installed on the system. The project yarn will not resolve a globally installed package. yarn and npm resolving a global package (sometimes in lieu of a package explicitly specified in the project manifest!) is the cause of much frustration and is just flat-out not a problem for yarn v3 projects.

Concerns:

• The small overhead inherent to updating the project's copy of yarn and plugins. It is not a huge hassle (it is a single command), but it is a thing that will need to be done from time-to-time.
• yarn v3 is stable but it's obviously not as battle-tested as its older brother.

Things I'd like to see explored (not blocking for this PR):

• A resolutions entry in package.json that resolves react to @wordpress/element.

[knowler]

I did a little research about Yarn 3 and very quickly learned about Corepack. Given that Sage 10 is a Node 16+ project and Corepack should be readily available, I am no longer opposed to using Yarn going forward. I am indifferent though, but that’s just because I haven’t used Yarn 2 or 3. It seems reasonable to stick with Yarn since most existing Sage users will be familiar with it.

Two questions I have for this PR:

1. From my understanding there are no node_modules in Yarn 2+? If so, then can we remove the reference to it in the README’s theme structure tree.
2. Now, more importantly, how well is Yarn 3 supported by CI tooling? I think if we are going to adopt Yarn 3, then we need to make sure that the major CI tools the community uses support it.

[benpsnyder]

I'm glad to lend insight as I have worked with Yarn v3 extensively with the patching capability (its awesome) and with CI caching. Will reply tonight.

Edit: also, Yarn v2 was an abomination

[QWp6t]

@knowler

1. From my understanding there are no node_modules in Yarn 2+? If so, then can we remove the reference to it in the README’s theme structure tree.

That's what nodeLinker: node-modules is for. It will cause Yarn to add node_modules directory for package resolution.

2. Now, more importantly, how well is Yarn 3 supported by CI tooling? I think if we are going to adopt Yarn 3, then we need to make sure that the major CI tools the community uses support it.

I'm not sure about the wider landscape, but notably the Dependabot team currently has no plans to support Yarn 2/3 (you'll have to view hidden comments to see the conversation, but it's all in there).

We ran into quite a few issues in Bud where it turned out that Yarn was the culprit. Our issues were related to https://github.com/roots/bud being a monorepo, and also possibly something to do with Typescript (at one point, builds were failing on my machine bare metal, but worked in the docker container, but failed on github, but worked bare metal for Kelly, and it turned out that reverting a Yarn update resolved everything). It definitely caused some headaches and I have mixed feelings about it in general, but also, I can't think of any issues we ran into that would directly affect Sage.

[knowler]

@QWp6t Thanks for explaining that. I guess we will need to use Renovate for dependency updates like Bud is if we merge this.

[retlehs]

if bud could work on npm instead of requiring yarn, i personally would prefer to drop use of yarn entirely in sage due to the current state of yarn

[retlehs]

after looking into this a bit more, i'm a little hesitant because:

• we'll have to move away from dependabot like @QWp6t mentioned. this isn't a dealbreaker, dependabot isn't perfect anyway, but it is a blocker in order to merge this PR.
• we'll have to include the yarn binaries in the repo (3mb) and update them over time 🙈

...but until bud supports npm, we do need to update the docs and README's for both bud and sage to reflect that yarn is a requirement to avoid confusion. per @kellymears, both yarn v1 and yarn v3 are supported. it doesn't seem right to recommend installing yarn v1 though, and it looks like the changes from this PR are required for v3.

🤔

this looks like something that we're gonna have to do. i just don't want to do it.