make work with pnpm and git, use pnpm-lock.yaml

[milahu]

part of issue https://github.com/ds300/patch-package/issues/35#issuecomment-691512252

error was

**ERROR** No package-lock.json, npm-shrinkwrap.json, or yarn.lock file.

install

to install my patched version of patch-package

pnpm install @milahu/patch-package-with-pnpm-support
npx patch-package

why is this PR still open?

the author of patch-package has abandoned the project, so nobody has write access to the repo. feel free to fork my fork

changes

• use pnpm-lock.yaml to get the resolvedVersion
• use pnpm to install the origin package (if pnpm-lock.yaml was found)
• workaround for pnpm symlinks
• add some debug output, also show the output of npm install etc (todo: add a --verbose flag to CLI)
• exclude lockfiles from diff. maybe exclude more files with patch-package --exclude '^dist/.*'
• fix some lint errors via tslint --fix and prettier --write (sorry for the diff noise)
• allow comment header in patch files, see my comment
• support pnpm's link:../../../some/relative/path
  • detect local git repos
    • patch is based on the origin/HEAD commit
    • use full commit hash as version for npm
    • as fallback, use version from local package.json
  • dont copy node_packages folders from local package

todo

are git patches applied correctly? we must install the exact commit

fixme

when we install packages from git (pinned to commit), patch-package will say Warning: patch-package detected a patch file version mismatch cos the patch version is the "declared" version from /package.json = commit hash = for example 8a5162785c58864ef65308c0f0d890cade0b407f and the "applied to" version ("resolved" version) is from node_modules/pkg/package.json = for example 1.0.0-canary.10 both versions (declared and resolved) could be stored in the comment header of the patch file

[milahu]

oof. tests pass locally but fail on ci cos (snapshots && snapshots.length) == null maybe the *.sh scripts run in posix shell?

edit: yepp, removed shopt -s expand_aliases

[worstpractice]

@milahu Awesome work! I was just about to look into this myself when I found your PR.

[PabloSzx]

@ds300 Is it possible to take a look into this PR? thank you

[milahu]

thx for the keepalive ; )

thing is, the main patch-package repo is slightly undermaintained ... i mean, my last patch (#252) was 3 lines, and that took 190 days to merge. looks like ds300 is the only maintainer and shifts the backlog only like 2 or 3 times per year : /

i never really came to use my patch, let me catch up on that:

install patch-package branch pnpm-support-2

cd /my/fancy/node/project
mkdir patched_packages
cd patched_packages
git clone --depth 1 https://github.com/milahu/patch-package.git --branch pnpm-support-2
cd patch-package
pnpm install
pnpm run build
cd ../../
node patched_modules/patch-package/dist/index.js some-node-package

tests

edit: github CI says tests passed 0__o let me run tests again ... edit 2: nope, still the same error:

# if you have some minutes to waste ...
pnpm run test
# jest --runInBand                                                 
# ts-jest[config] (WARN) Unable to find the root of the project where ts-jest has been installed.
# ts-jest[versions] (WARN) Version 4.2.4 of typescript installed has not been tested with ts-jest. If you're experiencing issues, consider using a supported version (>=2.7.0 <4.0.0). Please do not report issues in ts-jest if you are using unsupported versions.
# FAIL src/packageIsDevDependency.test.ts
#   ● packageIsDevDependency › returns false if package is a transitive dependency of a dev dependency
# [ may be related to my patch ]
# Test Suites: 1 failed, 35 passed, 36 total
# Tests:       1 failed, 5 skipped, 804 passed, 810 total
# Snapshots:   65 passed, 65 total
# Time:        438.918s

one problem i found: when /package.json says .dependencies.some-package = link:../../some/where then /pnpm-lock.yaml says .dependencies.some-package = 1.2.3 so the "resolved" version (from /pnpm-lock.yaml) is wrong and my "detect git version" logic in patch-package/getPackageResolution.js is not used -> added a workaround

another problem, also

when /package.json says .dependencies.some-package = link:../../some/where

... then that /package.json will obviously not work on other machines (not portable) and we would have to replace link:../../some/where with the git-version (or npm-version), that the patch is based on or straight-up throw a fatal error in patch-package when /package.json says link:../../some/where?

opinions on this? what is a "reasonable default" workflow to combine git and pnpm?

when i do pnpm install git+https://github.com/ds300/patch-package.git then node_modules/patch-package is not a git repo

when i do

(
  cd /tmp; git clone --depth 1 https://github.com/ds300/patch-package.git
  cd patch-package; pnpm install; pnpm run build
)
pnpm install /tmp/patch-package

then the resolvedVersion in /pnpm-lock.yaml is wrong (not the git version) and /package.json is not portable

so ... imho the only solution is to make pnpm clone the git repo by running something like pnpm install --git-clone git+https://github.com/ds300/patch-package.git this would clone with a default depth of 1, allowing to set --git-depth <depth>

ideally, following runs of pnpm install or pnpm remove should not change the git repo

[PabloSzx]

when /package.json says .dependencies.some-package = link:../../some/where

... then that /package.json will obviously not work on other machines (not portable) and we would have to replace link:../../some/where with the git-version (or npm-version), that the patch is based on or straight-up throw a fatal error in patch-package when /package.json says link:../../some/where?

opinions on this? what is a "reasonable default" workflow to combine git and pnpm?

I feel like using patch-package on a locally linked package is way too niche, and if it takes way too much work or maintenance in the future just to keep it working, it should just throw, I'm pretty sure the 99% of the usage of patch-package is just for npm packages.

[milahu]

it should just throw

agree, linked packages are not portable

I'm pretty sure the 99% of the usage of patch-package is just for npm packages.

... but some changes are non-trivial for example when i want to cherry-pick some unmerged PRs (i usually create PRs for my patches) but i want to publish my "private merge" branch via patch-package, not via git

my workflow can be solved with

1. in the npm project, install some-package from git. pin the version with the full commit hash we can install the latest version from npm, but then the diff is larger 2. in a separate folder, clone the git repo of some-package, make changes, branches, commits, ... 3. copy the patched files to the npm project's node_packages/some-package/ sample: rsync --archive /tmp/repo/some-package/ /tmp/project/node_packages/some-package/ 4. run patch-package some-package

step 3 could be automated with a file watcher, e.g. chokidar.watch

another detail (see next comment) the version of the patched package should be pinned in /package.json (single source of truth) so it works with all package managers at all times

[milahu]

done: restore the old getPackageVersion for file protocol (to fix tests), and for other protocols (http://some/file.tgz#sha1sumoftgzfile, etc.)

when the package is installed from git, patch-package will require to pin the version in package.json:

{ 
  "dependencies": {
    "some-package": "git://some/where#fullcommithash"
  }
}

todo: automate the version-pinning with a --pin-version flag for patch-package -> will edit package.json to pin the latest version

discuss: is this "version-pinning in package.json" too pedantic? should we "just create a patch" and complain later on version mismatch? currently applyPatches.ts will say Warning: patch-package detected a patch file version mismatch

todo: fix installedPackageVersion in applyPatches.ts -> should use the commit hash for git repos either from declaredVersion, or from git rev-parse HEAD

fixed: the --debug flag consumed the next argv replaced the unmaintained minimist argv parser with dashdash

tests: to test npm packages, we could run a local npm registry

done: add header comments to the generated patch files. sample:

# generated by patch-package 6.4.7 on 2021-05-12 20:07:53
#
# command:
#   npx patch-package --exclude '^test/stubs.*' @11ty/eleventy
#
# declared package:
#   @11ty/eleventy: github:11ty/eleventy#8a5162785c58864ef65308c0f0d890cade0b407f
#
# header comments also make room for custom "patch messages", like ...
#
# this patch will
#
#  * add feature xyz
#  * fix bug 123
#
diff --git a/node_modules/@11ty/eleventy/src/Engines/Txt.js b/node_modules/@11ty/eleventy/src/Engines/Txt.js
new file mode 100644
index 0000000..aef38ca
--- /dev/null
+++ b/node_modules/@11ty/eleventy/src/Engines/Txt.js
@@ -0,0 +1,34 @@
+// based on Html.js
+
+const TemplateEngine = require("./TemplateEngine");
...

todo: when package is a git repo, add commit messages as diff header comments

todo: when an old patch exists, prepend the old header comments to the new patch

done: add prepare script to auto-build on install from source

pnpm i github:milahu/patch-package#pnpm-support-2 # requires pnpm 6.2.2
npx patch-package

todo: emulate git status to preview what files will be changed or added -> exclude unwanted files before makePatch

[Jack-Works]

hi @ds300 can you review this? thanks!

[milahu]

so ... maintainer is gone, nobody wants to maintain a fork, so we must install patch-package from some hidden fork branch, cos we cant use patch-package to patch patch-package : D

pnpm i github:milahu/patch-package#pnpm-support-2 # requires pnpm 6.2.2
npx patch-package

its not perfect, but it works : )

[OmgImAlexis]

@ds300 can you please review this?

[schmod]

Does anybody know if the pnpm folks would be willing to incorporate this directly into pnpm (similar to Yarn 2)?

[Jack-Works]

Tracking issue in pnpm: https://github.com/pnpm/pnpm/issues/3077

[zheeeng]

Expect progress on this

[JessicaSachs]

Thanks for keeping this branch alive. It's a huge help. I did have to take away all of the wildcard selectors in my package.json and pin the exact versions, which isn't great, but at least it worked!

🤞 hopefully the maintainer can give this a quick look. Pretty please @ds300

[Jack-Works]

This works well in our project though I hope it can also support workspace so I don't have to install the package at the top level. Hope this can be merged soon

[milahu]

I did have to take away all of the wildcard selectors in my package.json and pin the exact versions

why? whats your package manager?

I hope it can also support workspace

PRs welcome

[JessicaSachs]

why? whats your package manager?

pnpm 6.13.0, this is w/ an extremely basic project. I think your PR should be merged without changes because right now nothing works. Was just writing it up in case someone else came across the workaround.

[milahu]

right now nothing works

cannot reproduce

cd $(mktemp -d)
npm init -y
pnpm i github:milahu/patch-package#pnpm-support-2 # requires pnpm 6.2.2
npx patch-package

ouput

patch-package 6.4.7
Applying patches...
No patch files found

env

$ npm --version
8.1.0

$ pnpm --version
6.22.2

[JessicaSachs]

Well, uhm, you gotta install a dependency and try to patch package it.

Steps to reproduce

1. Download patch-package from your nice PR branch
2. Install any dependency (by default uses wildcard notation)
3. Run patch-package

cd $(mktemp -d)
npm init -y
pnpm i github:milahu/patch-package#pnpm-support-2 # requires pnpm 6.2.2
npx patch-package
pnpm install lodash # Writes ^14.XX.XX to package.json
pnpm patch-package lodash

And you'll get...

➜  tmp.SLmTyTJs pnpm patch-package lodash  patch-package 6.4.7
• Creating temporary folder
Error: error: found wildcard version. package.json must pin the exact version of lodash in the format <package>@<major>.<minor>.<patch>

And the full stacktrace....

Like my comment says, once you remove the wildcard notation by swapping the lodash version from ^14.XX.XX => 14.XX.XX (like the error message says) the error disappears.

Versions:

pnpm 6.13.0
npm 6.14.13 # Also fails on 8.1.0
node v14.17.0 # Also fails on v16.14.0
OSX Big Sur, 11.3

[JessicaSachs]

Ohhh. I understand why you were annoyed.

right now nothing works.

with the existing repo/project on main. Your PR works fine, minus the need to pin all of the dependencies. Regardless of if you fix the bug I posted above, your PR at least gives some sort of workaround.

[milahu]

thanks for reporting, fixed

# binary release
pnpm install @milahu/[email protected]

# source release
pnpm i github:milahu/patch-package-with-pnpm-support
pnpm i github:milahu/patch-package#pnpm-support-2

note: some tests are failing

[airtonix]

what's the hold up here ?

[milahu]

patch-package != pnpm

what's the hold up here ?

patch-package is unmaintained. feel free to use my pnpm install @milahu/patch-package-with-pnpm-support and/or maintain your own fork of patch-package

if you allow upstream packages to block your workflow, youre doing something wrong

[weyert]

patch-package is unmaintained. feel free to use my pnpm install @milahu/patch-package-with-pnpm-support and/or maintain your own fork of patch-package

if you allow upstream packages to block your workflow, youre doing something wrong

Thank you! Any reason why you didn't merge it into main of your fork of the patch-package? :)

[milahu]

this PR is merged in https://github.com/milahu/patch-package-with-pnpm-support

[weyert]

Thank you! Switching over to your version :>

[wighawag]

Hey @milahu

thanks for your work

this PR is merged in https://github.com/milahu/patch-package-with-pnpm-support

I tried to use it in mono-repo setup but this does not work

if I install at root and execute pnpm patch-package <package-to-patch> it cannot find the package

If I instead install it on the sub folder, it cannot find pnpm-lock.yaml

[milahu]

thanks for reporting. pnpm workspaces (monorepos) are not implemented should be easy to do ... please dont wait for me, i have no time for this see also #277 and yarn workspaces and npmv7 workspaces

[caschbre]

@milahu I found a fork of your repo that adds the pnpm lock file detection for workspaces... in case that is something you could easily roll back into your repo.

https://github.com/Tellus/patch-package-with-pnpm-support

[milahu]

thanks @Tellus thanks @caschbre

published as @milahu/patch-package-with-pnpm-support version 6.4.9

[Tellus]

thanks @Tellus

You're perfectly welcome. I mulled issuing a pull request back then but wasn't sure if it was up to code standards.

Glad to have been of help :)

Thanks for the great work on the package in general!

[milahu]

bugfix for pnpm workspaces published as @milahu/[email protected] tell me when it breaks ; )

[milahu]

renamed to @milahu/patch-package

@milahu/patch-package-with-pnpm-support is deprecated

[weyert]

How do people use patch-package with pnpm for a dependency of a workspace package?

[Jack-Works]

Good news!! PNPM now has official support for yarn style patch!

https://github.com/pnpm/pnpm/issues/2675

We just need to wait for its next release. The benefit of the official support in PNPM is that you won't corrupt the global store (they're shared!)

Also thanks for @milahu, your PR helped us go through the hard time before the official support!

[zkochan]

pnpm has a pnpm patch command since v7.4