yarn
yarn copied to clipboard
Installing package from a branch with yarn.lock present installs old version of package
This is bug report and a follow-up to issue #2280.
Problem:
When installing from a git branch with a correct yarn.lock
file present, yarn installs an old version of the package instead of the most recent commit (HEAD
), as defined in yarn.lock
.
This can lead to inconsistent install states when when using a yarn.lock
file.
Steps to reproduce:
The following steps simulate a case where a package has been installed from the master
branch of a GitHub repository, and what happens when a user tries to install a new commit to master
using yarn.
-
Create a new repository with a single commit of a file
version_1.0
and push it to GitHub (in this exampleaskielboe/yarn-bug
). -
Create a
package.json
file with the following content:
{
"name": "yarn-bug",
"version": "1.0.0",
"dependencies": {
"yarn-bug": "askielboe/yarn-bug#master"
}
}
-
Run
yarn install
at the location ofpackage.json
: yarn correctly installs the package with a single filenode_modules/yarn-bug/version_1.0
. -
Create a new commit in
yarn-bug
adding a fileversion_2.0
, and push the commit to GitHub. -
Run
yarn install
again: yarn installs from cache without checking for new commits at the origin. This means that the yarn installation ofyarn-bug
will not include theHEAD
commit with the new fileversion_2.0
. -
Delete
node_modules
andyarn.lock
and runyarn install
: Withyarn.lock
gone, yarn now correctly pulls the latest commit and generates a newyarn.lock
file with the HEAD hash. -
Delete
node_modules
(but keepyarn.lock
) and runyarn install
: yarn now installs the initial commit again, instead of the HEAD commit referenced inyarn.lock
. Somehow yarn looks at its cache and picks a commit (by random?) to install instead of checking whether it is the HEAD, or the most recent pulled commit.
Notes & workarounds:
-
Running
yarn upgrade
after deletingnode_modules
will install the correct version of the package. -
Clearing the yarn cache by running
yarn cache clean yarn-bug
and thenyarn install
will install the correctHEAD
commit without the need to deleteyarn.lock
. -
Even when using
--frozen-lockfile
yarn still installs an old commit (ignoring the correct hash inyarn.lock
). This can lead to inconsistent state and critical issues in continuous integration, when requiring packages from a git branch.
The bug(s) can be reproduced using the following script: (replace askielboe/yarn-bug
with a reference to your own test-repository). Please read through the script before running it, as it creates new directories and clears the yarn cache: https://gist.github.com/askielboe/0c0bc3fdf88193bb17fc4fef5d332ad1
This bug report has been created using yarn 1.2.1 and node 8.7.0 on macOS 10.12.6.
Thanks, @askielboe, this is a great bug report. The core team will look a this eventually but they are very busy at the moment. Do you think you could spend some time debugging git-resolver.js and see why is it resolving to the wrong revision?
I'll take a look if I have time.
Meanwhile I've updated the bug report to reflect the fact that you have to do an install between git commits for the bug to arise. I believe it's due to the fact that yarn then keeps references to several commits in its cache, resulting in inconsistent installations.
I've also included a new script to reproduce the bug in this way.
I've done a bit of debugging. The yarn codebase is a bit opaque to me unfortunately, and I won't be able to spend more time this week, but here is what I've managed to find so far.
It seems that the issue appears when the package is fetched with a manifest where uid
is equal to the package version instead of the commit hash. This happens when fresh: false
, and makes BaseFetcher
create a folder in yarn cache with path $(yarn cache dir)/npm-yarn-bug-0.0.0
, excluding a commit hash.
When calling yarn install
, yarn will use the cached version in npm-yarn-bug-0.0.0
regardless of when it was created, ignoring the commit hash in yarn.lock
. This happens even if there are caches of the correct commit in the yarn cache (i.e. $(yarn cache dir)/npm-yarn-bug-0.0.0-13dba66e55f4b8b8259577f417d73aef1809c9d8
).
If the cache folder npm-yarn-bug-0.0.0
is deleted, yarn installs the correct version until a new npm-yarn-bug-0.0.0
folder is generated.
This is the manifest I get when yarn creates the non-hashed cache folder:
{ name: 'yarn-bug',
version: '0.0.0',
_reference:
PackageReference {
resolver: [Object],
lockfile: [Object],
requests: [Array],
config: [Object],
registry: 'npm',
version: '0.0.0',
name: 'yarn-bug',
uid: '0.0.0',
remote: [Object],
dependencies: [],
permissions: {},
patterns: [Array],
optional: false,
level: 0,
ignore: false,
incompatible: false,
fresh: false,
location: null },
_remote:
{ resolved: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
type: 'tarball',
reference: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
hash: '',
registry: 'npm' },
fresh: false }
And this is the manifest I get when yarn created a cache folder including a git hash (which results in the correct version getting installed):
{ name: 'yarn-bug',
version: '0.0.0',
_reference:
PackageReference {
resolver: [Object],
lockfile: [Object],
requests: [Array],
config: [Object],
registry: 'npm',
version: '0.0.0',
name: 'yarn-bug',
uid: '13dba66e55f4b8b8259577f417d73aef1809c9d8',
remote: [Object],
dependencies: [],
permissions: {},
patterns: [Array],
optional: false,
level: 0,
ignore: false,
incompatible: false,
fresh: true,
location: null },
_remote:
{ resolved: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
type: 'tarball',
reference: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
registry: 'npm',
hash: undefined },
fresh: true }
The main differences are:
- not working -> working
-
uid: '0.0.0'
->uid: '13dba66e55f4b8b8259577f417d73aef1809c9d8'
-
fresh: false
->fresh: true
-
hash: ''
->hash: undefined
#4722 seems related (CC @SEAPUNK)
it might be the exact same issue, but not 100% sure i'll keep my issue up just in case
@askielboe, that is a great analysis and a huge help, thanks a lot for digging that deep. It looks like the problem is in cache folder for git dependencies, maybe the name is using package.json version instead of git hash.
cc @arcanis as he knows more about Yarn/git integration.
More help getting to the bottom of this is always welcome.
I have the same issue, any updates for fixing this bug?
the issue is reproducible to this day :(
I have the same issue here also.
This is still happening with yarn version 1.15.2.
Same issue for me in the latest version (1.21.1). Any updates on this one?
Happens to me at 1.22.0 as well.
I go around it with yarn cache clean xxx && rm -rf node_modules/xxx && yarn --check-files
but I had it appearing over and over again, which is quite annoying I have to admit.
Are there any plans on fixing this? Otherwise I'll have to get back to NPM since Yarn literally it installs wrong dependencies on production.
From what I see, when the package.json refers to a branch, the commit for the current HEAD is stored in the yarn.lock, then when more commits are added, a yarn install always installed the saved commit instead of the new HEAD.
Any updates on fixing this? been living with constantly doing yarn upgrade my-package while in development.
Im also affected by this issue and experience the same behavior as @cah-briangantzler.
Since yarn 2 yarn upgrade
is replaced with yarn up
so I use the following workaround:
-
yarn up PACKAGE_NAME
-> this installs the latest version available in the npm repo - revert change version in
package.json
to the gitlab branch -
yarn install
It seems the only way to get things to work reliably is to use commit= instead of branch= :disappointed: OR to update the commit in the resolution
field of yarn.lock
Using yarn 3.2.0 and still having this issue. @raphaelschwinger 's workaround didn't work for me.
I believe this is a feature, not a bug. This is in fact the entire purpose of having a yarn.lock file: to ensure that every time you run yarn install
, it installs the same version of each dependency.
What would happen if yarn did not work this way?
- First time
yarn install
is run, yarn.lock points to commitaaaaaa
of package yarn-bug - App is tested and release to production
- Developer modifies package.json, e.g adding / upgrading unrelated dependency
- Developer runs
yarn install
- Yarn.lock now points to commit
bbbbbb
of package yarn-bug
So the package yarn-bug has now been inadvertently upgraded to a new commit that could potentially introduce bugs. This is the reason why versions numbers and commit hashes are locked down in yarn.lock, so that packages are only upgraded to a new version / commit when explicitly requested.
@pacaplan The thing is, if you define your dependency to target a branch and not a specific commit, I would expect it to actually updates to the HEAD of the branch at any install.
If I want to fix the version to a specific commit, I can use the commit hash on my dependency ("yarn-bug": "https://github.com/askielboe/yarn-bug.git#f3fb60b37da376cbe480b22587e73604d99f4f9e"
).
I feel like this is developer's responsibility to tell yarn to install from a commit or a branch. We know it's risky that it updates from the HEAD, but that's what expect anyway!
My solution to this problem was just to not use Git repos in Yarn/npm because it has some disadvantages I found out during my work on Open-Source projects:
- Some CI systems don't have Git installed
- When you transpile code, you will have to add the
dist
folder to the repo so that it will be in the installednode_modules
folder
I know it's not a fix to the actual problem but maybe you don't have to solve the problem after all 😃.
This is in fact a bug, not a feature. When dealing with branches and handling internal dependencies you want the immediate back and forth between the library and the project, having the resolution be stick to the commit should check if the commit is actual HEAD for the branch it intends to represent.