Local workspaces not being symlinked to node modules (when circular deps exist)

[milesj]

Do you want to request a feature or report a bug?

Bug report. I have this project called Beemo, which is a monorepo with workspaces/packages like @beemo/cli, @beemo/core, and @beemo/driver-*. Beemo is a configuration management layer, and as such, I have another NPM module called @milesj/build-tool-config, which depends on the same released version (0.13). Basically Beemo is using Beemo to configure itself, and technically I have a reference chain like so:

/packages
  /core (@beemo/core)
  /cli (@beemo/cli)
  /driver-* (@beemo/driver-*)
/node_modules
  /@milesj/build-tool-config
    /node_modules
      /@beemo/core (0.13)
      /@beemo/cli (0.13)
      /@beemo/driver-* (0.13)

When I yarn install, the dependent versions are hoisted to the root, like the example below. At this point, local /packages are not symlinked to /node_modules, as I assume Yarn believes that path is already taken. BUT, the @milesj/build-tool-config package doesn't depend on all @beemo packages, yet the packages that aren't depended on DO NOT get symlinked either (for example, @beemo/driver-mocha is not symlinked).

# hoisted
/packages
  /core (@beemo/core)
  /cli (@beemo/cli)
  /driver-* (@beemo/driver-*)
/node_modules
  /@beemo/core (0.13)
  /@beemo/cli (0.13)
  /@beemo/driver-* (0.13)
  /@milesj/build-tool-config

If I utilize nohoist to not hoist the @milesj/build-tool-config packages to the root, like the following, local packages STILL DO NOT symlink.

# nohoist
/packages
  /core (@beemo/core)
  /cli (@beemo/cli)
  /driver-* (@beemo/driver-*)
/node_modules
  / NO SYMLINKS
  /@milesj/build-tool-config
    /node_modules
      /@beemo/core (0.13)
      /@beemo/cli (0.13)
      /@beemo/driver-* (0.13)

Because of this, I'm unable to work on this project, or release new versions.

What is the current behavior?

Local /packages are not symlinked whatsoever, with hoisting, and without hoisting.

What is the expected behavior?

Local /packages should be symlinked to /node_modules. This used to work in Yarn 1.3, but seems to have broken in 1.5. This might be related to the new nohoist feature.

Please mention your node.js, yarn and operating system version.

Executable	Version
lerna --version	3.0.0-beta.3
npm --version	5.6.0
yarn --version	1.5.1
node --version	9.8.0

OS	Version
macOS Sierra	10.13.3

[connectdotz]

@milesj I clone your beemo repo and ran the yarn install, here is what I got from the project root:

$ pwd
xxx/beemo-master
$ ls -ld node_modules/@beemo/*
lrwxr-xr-x  1 vsun  staff   18 Mar 19 07:51 node_modules/@beemo/cli -> ../../packages/cli
lrwxr-xr-x  1 vsun  staff   19 Mar 19 07:51 node_modules/@beemo/core -> ../../packages/core
drwxr-xr-x  9 vsun  staff  306 Mar 19 07:46 node_modules/@beemo/driver-babel
lrwxr-xr-x  1 vsun  staff   28 Mar 19 07:51 node_modules/@beemo/driver-eslint -> ../../packages/driver-eslint
lrwxr-xr-x  1 vsun  staff   26 Mar 19 07:51 node_modules/@beemo/driver-flow -> ../../packages/driver-flow
lrwxr-xr-x  1 vsun  staff   27 Mar 19 07:51 node_modules/@beemo/driver-mocha -> ../../packages/driver-mocha
lrwxr-xr-x  1 vsun  staff   30 Mar 19 07:51 node_modules/@beemo/driver-prettier -> ../../packages/driver-prettier
lrwxr-xr-x  1 vsun  staff   32 Mar 19 07:51 node_modules/@beemo/driver-typescript -> ../../packages/driver-typescript

the @beemo pcakges are symlink all right, including @beemo/driver-mocha. Then looking at the @milesj/build-tool-config:

$ ls -ld node_modules/@milesj/build-tool-config/node_modules/@beemo/*
drwxr-xr-x  8 vsun  staff  272 Mar 19 07:51 node_modules/@milesj/build-tool-config/node_modules/@beemo/cli
drwxr-xr-x  8 vsun  staff  272 Mar 19 07:51 node_modules/@milesj/build-tool-config/node_modules/@beemo/core
drwxr-xr-x  7 vsun  staff  238 Mar 19 07:51 node_modules/@milesj/build-tool-config/node_modules/@beemo/driver-eslint
drwxr-xr-x  7 vsun  staff  238 Mar 19 07:51 node_modules/@milesj/build-tool-config/node_modules/@beemo/driver-flow
drwxr-xr-x  7 vsun  staff  238 Mar 19 07:51 node_modules/@milesj/build-tool-config/node_modules/@beemo/driver-jest
drwxr-xr-x  7 vsun  staff  238 Mar 19 07:51 node_modules/@milesj/build-tool-config/node_modules/@beemo/driver-prettier

Indded the @beemo packages there are not symlink, asking yarn to explain:

$ yarn why @beemo/core
yarn why v1.5.1
[1/4] 🤔  Why do we have the module "@beemo/core"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
=> Found "@beemo/[email protected]"
info Reasons this module exists
   - "_project_" depends on it
   - Hoisted from "_project_#@beemo#core"
info Disk size without dependencies: "5.11MB"
info Disk size with unique dependencies: "16.63MB"
info Disk size with transitive dependencies: "29.34MB"
info Number of shared dependencies: 91
=> Found "@milesj/build-tool-config#@beemo/[email protected]"
info This module exists because "_project_#@milesj#build-tool-config" depends on it.
info Disk size without dependencies: "112kB"
info Disk size with unique dependencies: "11.63MB"
info Disk size with transitive dependencies: "24.34MB"
info Number of shared dependencies: 91
✨  Done in 3.71s.

looks like there are 2 different versions: @milesj/build-tool-config is using 0.12.0, while the workspace version is 0.14.0. yarn will not share a package across different versions.

This seems right to me, not quite sure what the problem you were seeing... can you elaborate?

[milesj]

Weird, this seems to be working again after I ran yarn cache clean (maybe a bad cache?). Before then, I couldn't get symlinking working at all, so I had to resort to a custom bash script as to not block me.

I'll leave this open for a bit longer to see if I can replicate.

[jonaskello]

I had an issue where yarn did not link any local packages and yarn cache clean solved it. Thanks to @milesj !

[sebinsua]

I'm almost certain that I ran into this.

I ended up fixing it using some resolutions with the package names of my repository pointed towards versions of *.

However, at some point I would like to create a test case and try to troubleshoot it better...

[milesj]

It still occurs for me, but very randomly. I have yet to figure out a pattern on why it happens.

[DVLP]

same problem

[davidtai]

I have a work around for some of the the issue for OSX.

If yarn cannot find your link after you yarn link, check your ~/.config/yarn/link/ directory to make sure the symlink in there actually works. My yarn generating broken symlinks. The solution that worked for me was manually adding symlinks using the ln -s <your package dir> <symlink>.

If you use the @org notation, then you need to create a folder for the org and create the symlink under that.

[trusktr]

I'm having this issue. All my packages depend on one package using "^0.10.0" and the dependency that they depend on has version 0.10.0 in its package.json. When I run yarn install, the dependency package never gets linked to other packages, and instead it installs the dependency from npmjs.com.

When the dependency package was at 0.9.0, and all the dependent packages depended on ^0.9.0, linking worked fine. When I modified everything to 0.10.0, yarn.lock was updated by yarn install, but links stopped happening.

[trusktr]

I had to delete yarn.lock, then linking started working again. But this workaround risks things breaking if indirect dependencies from npm get updated and there's a breaking change.

Ideally modifying the versions in package.json of all workspaces should be enough, with subsequent yarn install making the minimal modifications to yarn.lock.