[rush] fix: allow specifying linked deps with PNPM

[sammarks]

This is a continuation of my comments from #876 (and potentially also relates to #1719).

I'm trying to accomplish linking to packages outside the scope of the current rush project (in my case, I have one monorepo that contains all of the core functionality of a framework I use across multiple separate products, which each have their own corresponding monorepos with the packages specific to their functionality).

I've come up with what seems to be working pretty well (at least in my local development, over the last few weeks). That involves having a links.json file in the config directory to specify which packages should be linked from which other projects.

Then, the pnpmfile.js reads this JSON file and modifies the dependencies whenever it sees them, inserting link: references to the other packages.

This works very well! The problem is Rush, whenever generating the shrinkwrap and cache files to keep track of which packages have updated in between calls, fails to generate these files properly when there are packages with link: version specifiers.

So this PR basically just updates the respective code to ignore those packages.

The only caveat I've experienced thus-far, is whenever you "unlink" a package (that is, remove it from the links.json file), you have to run rm -rf */*/node_modules && rush update --full, otherwise PNPM / Rush still think the package should be installed as a link and doesn't remove the link. Not sure if this is an issue with Rush, or an issue with PNPM (my guess would be it's an issue with PNPM not respecting the updates to the package.json).

For a concrete example, I have a PNPM file that looks like the following:

'use strict'

const fs = require('fs')
const path = require('path')

let linkedPackages = {}
const LINKS = path.join(__dirname, '../config/links.json')
if (fs.existsSync(LINKS)) {
  const contents = fs.readFileSync(LINKS).toString()
  linkedPackages = JSON.parse(contents)
}

module.exports = {
  hooks: {
    readPackage,
    afterAllResolved,
  },
}

function afterAllResolved(lockFile, context) {
  for (const importerKey of Object.keys(lockFile.importers)) {
    // Check to make sure it starts with ../../ because that means it's probably
    // inside the current project.
    if (importerKey.startsWith('../../')) {
      for (const root of Object.keys(linkedPackages.roots || {})) {
        for (const linkedPackageName of Object.keys(linkedPackages.roots[root] || {})) {
          const finalPath = path.join('../..', root, linkedPackages.roots[root][linkedPackageName])
          context.log(`(lockfile) ${linkedPackageName} → link:${finalPath}`)
          lockFile.importers[importerKey].dependencies[linkedPackageName] = `link:${finalPath}`
        }
      }
    }
  }

  return lockFile
}

const DEP_KEYS = ['dependencies', 'devDependencies']
function readPackage(packageJson, context) {
  for (const depKey of DEP_KEYS) {
    for (const root of Object.keys(linkedPackages.roots || {})) {
      for (const linkedPackageName of Object.keys(linkedPackages.roots[root] || {})) {
        if (packageJson[depKey][linkedPackageName]) {
          const finalPath = path.join('../..', root, linkedPackages.roots[root][linkedPackageName])
          context.log(`${linkedPackageName} → ${finalPath}`)
          packageJson[depKey][linkedPackageName] = `${finalPath}`
        }
      }
    }
  }

  return packageJson
}

And a links.json file that looks like the following:

{
  "roots": {
    "../other-project": {
      "@other/project-react": "react/project-react"
    }
  }
}

This would register link:../other-project/react/project-react as the version specifier for any reference to a @other/project-react package.

Running this pnpmfile with the latest version of Rush, I get errors about not being able to modify the shrinkwrap file or not being able to find the package inside the shrinkwrap file.

Running this pnpmfile with a built-version of this PR, those errors go away and subsequent installs appear to be working fine, minus the issue mentioned above about the steps required whenever removing those dependencies.

---

With that said, not sure if this is the most appropriate place for this code because arguably my use-case seems like a very specific workaround to a much broader problem, but I at least think introducing a workaround like this is a start while the details are fleshed out about the rush bridge idea.

[iclanton]

This is really interesting. Can you link to an example repo that uses this functionality so we can try it out?

I think if this functionality were to be built into Rush directly, we'd want this feature to be more fleshed out and completely driven by config files (i.e. - the repo maintainer wouldn't need to insert code into pnpmfile.js manually).

I wonder if this could be achieved with a companion tool that uses rush-lib.

@octogonz - Thoughts?

[octogonz]

This is really interesting. Can you link to an example repo that uses this functionality so we can try it out?

+1 Please show an example.

Also, I'm curious how this relates to "useWorkspaces": true in rush.json. Are workspaces required? (Or not supported?)

I think if this functionality were to be built into Rush directly, we'd want this feature to be more fleshed out and completely driven by config files (i.e. - the repo maintainer wouldn't need to insert code into pnpmfile.js manually).

I wonder if this could be achieved with a companion tool that uses rush-lib.

I like the idea of building it directly into Rush. It is a very common request, and right now we don't have a good answer. The way I personally handle this is by manually creating symlinks, using a procedure that is too complicated to document for everyday usage.

@sammarks If you could provide an example repo and some step-by-step instructions for how you link/unlink projects, that would be helpful. I'd like to help complete the design so we can make it a fully supported/documented Rush feature, but my own time is spread very thin right now, so if you have some concrete examples we can study that could save a lot of time.

[sammarks]

Sorry for the delay here gentlemen! I've whipped up a quick sample repository for you to peruse:

https://github.com/sammarks/rush-demo-linked-packages

All of the instructions for getting it setup & steps for linking / unlinking packages are included in there. Like I said, right now, the process is a little cumbersome, but it's at least a good step, I feel, in the right direction.

[slavafomin]

I've created a similar tool: super-rush for exactly this purpose. Would love to see this implemented in Rush natively, because we have a lot of separate monorepos in our organization that would benefit from the ability to join together.