CLI functionality related to aggregator preventing yarn commands 1.19-1.22 - Reverted to 1.16

[DavideDaniel]

<!-- *Before creating an issue please make sure you are using the latest version of yarn.* --> ^ Problems started with upgrading to 1.19 and then snowballed into 5 hours of hell trying to figure out what went wrong.

Do you want to request a feature or report a bug? error An unexpected error occurred: "expected workspace package to exist for \"rimraf\"" It started with some dev dependency in a workspace repo and after messing around with nohoist settings I started noticing that the same modules keep coming up. Rimraf, inquirer, react-dev-tools.. generally things that were listed many times over in transitive dependencies. Read through many different similar issues regarding symlinks but ended up being unable to add new modules through cli which was the problem I was trying to address for automation.

What is the current behavior?

These commands do not seem to work in version v1.19.2 through v1.22.0-20191201.1637:

yarn workspace @scope/A add uuid lodash.get --dev cd packages/A && yarn add uuid lodash.get --dev yarn workspace @scope/A add js-cookie@^2.2.1 -D cd packages/A && yarn add lodash.get@^2.2.1 --dev

Tried yarn cache clear, removing node_modules, reinstalling yarn to no avail.

If the current behavior is a bug, please provide the steps to reproduce.

Sorry, work repos.

What is the expected behavior? Workspaces command and yarn add should work from cli.

Please mention your node.js, yarn and operating system version. Tested on the following:

Node: v8.9.4 v8.15.1 v10.11.0 v10.16.0 v10.16.2

Yarn: v1.19.2 v1.21.0 v1.22.0-20191201.1637

OS: MacOS Mojave 10.14.6

[DavideDaniel]

For anyone that comes looking for a quick fix.. I was able to get back to a working state by installing v16.0 https://raw.githubusercontent.com/karlhorky/homebrew-core/f8bc263f93dcfd1cc50c98976a8503c8a7cb032f/Formula/yarn.rb via homebrew. I'm gonna try seeing if the update with 1.17.4 works and go as far up as I can without losing functionality.

~I was able to go up to 1.17.4 but 1.19 and up the same issues come up.~

1.17.4 continues to have lots of symlink issues which were addressed in 1.19 but is not usable currently for me. 1.16 seems more stable but is not secure... sigh. Please let me know how I can help. Need a break and will check in later.

[DavideDaniel]

A clue about symlink issues while running --verbose. The uuid module keeps missing even as far back as 1.16. The common pattern right now appears to be modules that are used in many other modules but are listed as a peerDependency in a workspace can run into issues with workspace aggregator during certain cli commands. This could be because a lot of commonly used modules are listed as peerDependencies vs dependencies, following advice from https://yarnpkg.com/blog/2018/04/18/dependencies-done-right/.

[DavideDaniel]

From https://github.com/yarnpkg/yarn/issues/6953 I'm wondering if 2.0 will address this with the above.. but still unsure why this became a bigger problem past 1.16. Is it because changes to 1.16 are being made with the same lens applied for code going in for 2.0? If so, should we just consider pinning our version of yarn and wait for 2.0 if we are using workspaces?

[danoc]

I think we're running into this issue at @thumbtack. In particular, this affects us when trying to upgrade a package within a workspace.

This snippet should reproduce the issue:

git clone [email protected]:thumbtack/thumbprint.git && cd thumbprint && yarn && cd packages/thumbprint-react && yarn upgrade @thumbtack/thumbprint-tokens --latest

This is the error you'll see:

error An unexpected error occurred: "expected workspace package to exist for \"@babel/plugin-transform-dotall-regex\"".
info If you think this is a bug, please open a bug report with the information provided in "/Users/<redacted>/thumbprint/packages/thumbprint-react/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/upgrade for documentation about this command.

[danoc]

Confirmed that downgrading Yarn to 1.16.0 works. Thanks for digging into this, @DavideDaniel. Saved me lots of time!

Here are downgrade instructions for those that installed with Homebrew: https://github.com/thumbtack/thumbprint/issues/592#issuecomment-565564993

[troywweber7]

Downgrading worked for me. But I'd definitely like to see this fixed in the latest version of yarn so I'm not lagging behind by 5 minor versions... yikes!

[iplus26]

Downgrading to v1.19.0 worked for me. Looking forward to fix this~

[roni-frantchi]

Downgrading to v1.19.0 worked for me. Looking forward to fix this~

Same here.

[amirburbea]

Any update on this?

In my lerna monorepo I have some packages (like husky and lint-staged) installed at the root

(My .yarnrc file has --ignore-workspace-root-check true)

When I try to upgrade packages (using yarn upgrade-interactive --latest) I always get Invariant Violation: expected workspace package to exist for "" where the quoted portion includes the name of the root package. Downgrading to yarn 1.17.0 has no such issue

[troywweber7]

I just want to hop in and clarify my situation, in case it helps anyone else.

Failing Command

The commands that were failing for me were of the yarn workspace ${workspace} add ${package} variety.

Working Yarn Version

The indicated command is FAILING for me when using yarn=1.19.2-1 is SUCCEEDING for me when using yarn=1.19.1-1.

Install Older Yarn

First, follow the directions here: https://yarnpkg.com/lang/en/docs/install/#debian-stable

However, when it comes to the installation command, sudo apt update && sudo apt install -y --no-install-recommends yarn, you'll want to change yarn to yarn=${version}. You can find appropriate versions by doing apt-cache policy yarn.

Once you install a yarn version, you may also want to consider doing a sudo apt-mark hold yarn until this is resolved in later versions, otherwise apt upgrade will update your yarn again. You can confirm it is held by doing apt-mark showhold.

[narwold]

We are experiencing the same issue since updating to 1.21.1. Our message is as follows:

An unexpected error occurred: "expected workspace package to exist for \"@babel/traverse\"".

This is in a monorepo. When we downgrade to 1.19.0, the problem goes away. We don't want to do that, however, because of this recent security bulletin:

https://blog.npmjs.org/post/189618601100/binary-planting-with-the-npm-cli

Any suggestions?

[StephenHaney]

Also experiencing this, here's our info – happy to provide anything else that might be useful.

Basic workspace monorepo, not using lerna.

Yarn version: 1.21.1

In: yarn add ts-jest -D

Out: error An unexpected error occurred: "expected workspace package to exist for \"string-width\"".

string-width is there in my node_modules. I've tried with both string-width in the main workspace node_modules and nohoisted.

When I downgrade to 1.16.0 I can complete the install.

[GeorgeTaveras1231]

My team is also experiencing this issue with version 1.21.1. However, as some other folks noted, all install commands work on prior versions (1.19.1 in our case)

This is blocking us from addressing the security issue documented here:

https://snyk.io/blog/understanding-filesystem-takeover-vulnerabilities-in-npm-javascript-package-manager/

[AndreyChernykh]

The same issue for our team.

yarn 1.21.1 node v10.15.1

[dmfilipenko]

Invariant Violation: expected workspace package to exist for "core-js-compat" yarn 1.21.1 node v13.2.0

[maiertech]

Same issue with yarn 1.21.1 and node v12.14.1.

[IskanderMustafin]

Same here.

yarn --version
1.21.1

node --version
v12.9.0

[DalderupMaurice]

same here

node -v && yarn -v
v10.15.3
1.21.1

[aryzing]

For those looking for a solution that doesn't involve changing the version of yarn you're using, editing package.json to include the version of the package you want and then running the yarn command again works just fine.

I was having a similar issue when running yarn workspace <workspace> add <pkg>, and resorted to manually amending that package's package.json and running yarn. Idea from this post.

[GeorgeTaveras1231]

@aryzing thanks for that suggestion and sharing a link to that post. However, in our team with use the --frozen-lockfile flag by default (by adding it to our .yarnrc file) to avoid accidental changes to dependencies. It seems like we would need to disable that feature in order to manually edit package.json files -- and as a result, risk accidental changes to the dependencies across all workspaces, unless I'm missing something.

[codyfyi]

Here is a stack trace (node v12.14.0 , yarn v1.21.1):

verbose 2.685597913 Invariant Violation: expected workspace package to exist for "eslint-plugin-react"
    at invariant (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:2314:15)
    at _loop2 (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:94898:9)
    at PackageHoister.init (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:94957:19)
    at PackageLinker.getFlatHoistedTree (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:48743:20)
    at PackageLinker.<anonymous> (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:48754:27)
    at Generator.next (<anonymous>)
    at step (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:310:30)
    at /usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:328:14
    at new Promise (<anonymous>)
    at new F (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:5301:28)
error An unexpected error occurred: "expected workspace package to exist for \"eslint-plugin-react\"".

This appears to be a bug in the PackageHoister. See: https://github.com/yarnpkg/yarn/blob/c43f66d14b1baedc7f50055dd7c997b32be0f8b1/src/package-hoister.js#L855-L872

[Meemaw]

This is a serious issue as it effectively prevents any package from being installed.

Any update on when this is supposed to be fixed?

[DalderupMaurice]

This is a serious issue as it effectively prevents any package from being installed.

Any update on when this is supposed to be fixed?

you can manually adjust the package.json and run yarn install. The only sad part about this is that you have to sort the packages yourself from A-Z when adding new packages 😂

[troywweber7]

Any update from yarn maintainers on whether this issue is even seen or recognized?

[friederbluemle]

Just updated to 1.22.0 via Homebrew and ran into this issue as well...

[pedroSoaresll]

Just updated to 1.22.0 via Homebrew and ran into this issue as well...

sorry, to me the version 1.22.0 don't solve the problem. :disappointed:

[steffeli]

I've run into this problem as well a couple of weeks ago. Monorepo using Lerna and yarn workspaces. All of a sudden adding packages causes An unexpected error occurred: "expected workspace package to exist for \"@typescript-eslint/eslint-plugin\"". Upgraded to 1.22.0, problem still persist.

Right now we have to manually add packages to the package.json and run yarn install.

[vleandersson]

Upgraded from 1.19.x to 1.22.0 and ran into this issue. yarn cache clean solved the issue for me.

[jpaas]

I just tried again to upgrade from 1.16.x. This time to 1.22.0. The problem still exists for me. yarn cache clean did not help.

[webmaster128]

I get the same error when yarn adding something to a package of a monorepo:

error An unexpected error occurred: "expected workspace package to exist for "@storybook/react"".

@storybook/react is installed in the monorepo root's package.json in the devDependencies. When I change the monorepo root's devDependencies to dependencies, the issue disappears. I don't think you should do that but maybe it helps understanding and fixing the problem.