rushstack icon indicating copy to clipboard operation
rushstack copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

[rush] Add split workspace feature to manage a secondary pnpm workspace

Open chengcyber opened this issue 2 years ago • 30 comments

📌 This PR is marked as "Draft" because it solves a temporary problem at our company. We will publish a preview release, and use it internally for some time, then make a decision whether to officially integrate it into Rush.

📢 Update: This feature has been superseded by the upcoming Rush subspaces feature. The split workspace PR will be retired after subspaces has shipped.

Summary

This PR implements split workspace feature, which is originally proposed at https://github.com/octogonz/rush-split-workspace-prototype.

Proposal

  • Patch the Rush software so that rush install automatically generates and installs the secondary workspace, using "splitWorkspace": true to recognize the projects that have not been migrated yet, and thus should be part of the "old" workspace.
  • rush install will also inject the .pnpmfile.cjs workaround to rewrite workspace: -> link: as appropriate
  • Patch the Rush software so that rush build considers processes all projects (a, b, c, d) as if they belong to a single workspace.

Context:

This feature aims to help those monorepo, who is managed by pnpm workspace and may have some ready to deprecated settings such as shared-workspace-lockfile=false, migrates into Rush.js. Ideally, it's easier to migrate all at once. but the challenges are:

  1. Trying to validate everything all at once would be difficult to coordinate between all the different teams, especially when the test coverage is not great.
  2. We can not stop development in the main branch while that's underway.

So, the splitWorkspace comes into rescue, helping gradually migrating projects into Rush.js world.

Details

  1. Add splitWorkspace: boolean field to specify a rush project under secondary workspace
  2. When rush update. Besides normally install the workspace projects, install for another pnpm workspace created under common/temp-split, containing split workspace projects.
  3. The split workspace are syncing common/config/rush/.npmrc-split-workspace as .npmrc. User can setup pnpm configurations here, such as shared-workspace-lockfile=false. The generated pnpm lockfile are copied into common/config/rush/split-workspace-pnpm-lock.yaml to be git committed.

Note:

Rush workspace project can not depends on splitWorkspace project, while splitWorksapce project can depends on Rush workspace projects, and Rush.js rewrites workspace protocol into link protocol during pnpm install for split workspace via specifing a shimmed global-pnpmfile.cjs.

Tasks

The task is all about make splitWorkspace working with current rush commands:

  • [x] rush update
    • [x] pnpmfileShim
    • [x] project shrinkwrap
    • [x] rush purge
  • [x] rush add
  • [x] selecting subset projects
    • [x] rush build --to foo
    • [x] rush build --from foo
    • [x] rush install --to foo
  • [x] rush deploy
  • [x] rush publish

How it was tested

  1. [x] make everything works fine in a demo project at https://github.com/chengcyber/rush-split-workspace-demo.
  2. [x] test in an internal repo, which is currently a pnpm workspace managed monorepo.

chengcyber avatar Jun 17 '22 04:06 chengcyber

🚀 A prerelease of this branch is published as version 5.75.0-pr3481.1 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

octogonz avatar Jul 28 '22 09:07 octogonz

Update: ensureConsistentVersions ignores split workspace projects for now.

chengcyber avatar Jul 30 '22 02:07 chengcyber

2022/8/16 Updates:

  1. Introduce "normalize-path" to fix the relative path in Windows platform, thanks to @EscapeB
  2. Fix "rush update" run by "rush add --package ", updating lockfile correctly.

chengcyber avatar Aug 16 '22 11:08 chengcyber

2022/8/18 Update:

improve error message when rush install fails with split workspace projects

image

chengcyber avatar Aug 18 '22 03:08 chengcyber

🚀 A prerelease of this branch is published as version 5.75.0-pr3481.3 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

octogonz avatar Aug 18 '22 07:08 octogonz

2022/8/19 Update:

Fix: rush add in a split workspace no longer blocked by ensureConsistentVersion policy

chengcyber avatar Aug 19 '22 04:08 chengcyber

🚀 A prerelease of this branch is published as version 5.75.0-pr3481.4 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

octogonz avatar Aug 22 '22 21:08 octogonz

2022/8/31 Update:

Merge the latest main branch (SHA: e434009246d37e38c8c45e9ed2c0754ba83d7b97) @microsoft/[email protected]

chengcyber avatar Aug 31 '22 13:08 chengcyber

🚀 A prerelease of this branch is published as version 5.77.2-pr3481.5 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

octogonz avatar Sep 01 '22 02:09 octogonz

🚀 A prerelease of this branch is published as version 5.78.0-pr3481.6 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

octogonz avatar Sep 21 '22 20:09 octogonz

Code updates 2022/9/29:

merged the latest main branch (SHA: a142f4608c1cf92fc937ff386d5db033412873ed),[email protected],

chengcyber avatar Sep 29 '22 11:09 chengcyber

Code updates 2022/10/31:

  • merged the main branch (SHA: https://github.com/microsoft/rushstack/commit/2ff426c64869457ade3e8f731efd69e8bdd3f098), [email protected]
  • suppress phantom folders warning to avoid confusion in teams.

chengcyber avatar Oct 31 '22 11:10 chengcyber

🚀 A prerelease of this branch is published as version 5.82.1-pr3481.7 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

octogonz avatar Nov 01 '22 03:11 octogonz

Code Updates:

  • remove aws/azure build cache plugin
  • merge the main branch (07b43995f8fd8c3228aae9b52436fe31bb5b897b)

chengcyber avatar Nov 11 '22 03:11 chengcyber

🚀 A prerelease of this branch is published as version 5.83.0-pr3481.9 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

octogonz avatar Nov 11 '22 05:11 octogonz

Update: merge the latest main branch, rush 5.86.0 SHA(b05c43e0accc66c5d3ac8a871f5d55eded248831)

chengcyber avatar Nov 30 '22 09:11 chengcyber

@chengcyber In the release branch, I had to fix a lint error:

[eslint] Warning: libraries/rush-lib/src/logic/base/BaseInstallManager.ts:272:54 - (@typescript-eslint/no-unused-vars) 'npmrcHash' is assigned a value but never used.
Encountered 1 warnings:
  [eslint] libraries/rush-lib/src/logic/base/BaseInstallManager.ts:272:54 - (@typescript-eslint/no-unused-vars) 'npmrcHash' is assigned a value but never used.

octogonz avatar Dec 01 '22 17:12 octogonz

🚀 A prerelease of this branch is published as version 5.86.0-pr3481.10 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

octogonz avatar Dec 01 '22 17:12 octogonz

Update 2022/12/10:

  • fix rush add dependencyAnalysis
  • merge the latest main branch (SHA: f6bcc624215e2eb3ffe079ff37888bf7ae79353c)

chengcyber avatar Dec 10 '22 01:12 chengcyber

🚀 A prerelease of this branch is published as version 5.86.0-pr3481.11 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

octogonz avatar Dec 10 '22 07:12 octogonz

🚀 A prerelease of this branch is published as version 5.86.0-pr3481.12 of the following packages (incorporating Ian's fix from PR https://github.com/microsoft/rushstack/pull/3837):

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

@chengcyber

octogonz avatar Dec 13 '22 03:12 octogonz

🚀 A prerelease of this branch is published as version 5.86.0-pr3481.14 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

@chengcyber

octogonz avatar Dec 22 '22 00:12 octogonz

🚀 A prerelease of this branch is published as version 5.90.0-pr3481.16 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

@chengcyber

octogonz avatar Jan 30 '23 07:01 octogonz

2023/3/6:

Merge the latest main branch (83db1b4889e7958808eb2cd43991e19b4c6d1023)

chengcyber avatar Mar 06 '23 04:03 chengcyber

🚀 A prerelease of this branch is published as version 5.93.1-pr3481.17 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

@william2958

octogonz avatar Mar 07 '23 05:03 octogonz

🚀 A prerelease of this branch is published as version 5.97.1-pr3481.18 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin
  • @rushstack/rush-http-build-cache-plugin ✨
  • @rushstack/rush-redis-cobuild-plugin ✨

This release includes the features from PR https://github.com/microsoft/rushstack/pull/3949 as of 9c32bf8aac1bb3490c78406085377b453264e43b.

@chengcyber

octogonz avatar Apr 22 '23 19:04 octogonz

🚀 A prerelease of this branch is published as version 5.100.1-pr3481.19 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

NOTE: Unlike 5.97.1-pr3481.18, this release does NOT include the Cobuilds feature from PR https://github.com/microsoft/rushstack/pull/3949

@chengcyber

octogonz avatar Jun 24 '23 05:06 octogonz

🚀 A prerelease of this branch is published as version 5.107.1-pr3481.20 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

This release includes Custom Tips, resolutionMode setting for pnpm-config.json, and the official release of Cobuilds.

@chengcyber @theJiawen @william2958 @g-chao

octogonz-proxy avatar Sep 22 '23 08:09 octogonz-proxy

🚀 A prerelease of this branch is published as version 5.109.1-pr3481.21 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

Relevant fixes:

  • Update the oldest usable Node.js version to 14.18.0, since 14.17.0 fails to load
  • Fixes for Custom Tips feature
  • Add autoInstallPeers setting in pnpm-config.json
  • (IMPORTANT) After upgrading, if rush install fails with ERR_PNPM_LOCKFILE_CONFIG_MISMATCH,
    please run rush update --recheck

@chengcyber @theJiawen @william2958 @g-chao

octogonz avatar Oct 10 '23 20:10 octogonz

🚀 A prerelease of this branch is published as version 5.109.1-pr3481.22 of the following packages:

  • @microsoft/rush
  • @microsoft/rush-lib
  • @rushstack/rush-sdk
  • @rushstack/rush-amazon-s3-build-cache-plugin
  • @rushstack/rush-azure-storage-build-cache-plugin

Relevant fixes:

  • feat: precisely generate project shrinkwrap when filtered install https://github.com/microsoft/rushstack/pull/3481/commits/b5ba6967994b1329db77f445b453d786da460a55

@chengcyber @theJiawen @william2958 @g-chao

octogonz avatar Feb 20 '24 01:02 octogonz