turbo icon indicating copy to clipboard operation
turbo copied to clipboard

[turborepo] lockfile changes causes all packages to be "changed" when using `--filter=[HEAD~1]`

Open dan-cooke opened this issue 2 years ago • 12 comments

What version of Turborepo are you using?

1.6.3

What package manager are you using / does the bug impact?

Yarn v1

What operating system are you using?

Mac

Describe the Bug

When any change is added to yarn.lock (even whitespace), running npx turbo run <command> --filter=[HEAD~1] causes the command to be run in every package.

Expected Behavior

I know this is probably a hard one to fix.

But I would prefer if turbo would parse the yarn.lock file and determine if the changed package affects a workspace before executing the command.

To Reproduce

  1. Create a new turbo repo
  2. Make a change to the yarn.lock (even a whitespace is fine)
  3. Commit this change
  4. Run npx turbo run build --filter=[HEAD~1]

Reproduction Repo

No response

dan-cooke avatar Nov 29 '22 11:11 dan-cooke

I'm going to have a look at fixing this - beacuse IMO this is a critical flaw in turborepo. It's very strange especially because the docs seem to acknowledge this as an issue with monorepos .

So you'd think this would be a core pillar of the design of turborepo.

Right now when I add any package to my repo - I have to re-deploy 10 apps which can take my GHA build up to 45 minutes (not to mention the potential outage of re-deploying 9 unchanged services)

dan-cooke avatar Dec 09 '22 18:12 dan-cooke

@dan-cooke did you find any solution or a workaround for this issue ? I have the same problem.

marwenguesmii avatar Dec 30 '22 14:12 marwenguesmii

@Marwennnne there is no solution or workaround - I have not got round to fixing it in turbo repo yet. I’m just taking the hit in CI runtime

dan-cooke avatar Dec 30 '22 14:12 dan-cooke

I appreciate this isn't a helpful addition but we're also running into this problem. We're having to re-build and re-deploy all of our apps whenever a dependency is added or changed in any of them.

kieranm avatar Jan 01 '23 15:01 kieranm

Issue appears also on yarn v3.2.4

izifortune avatar Jan 19 '23 16:01 izifortune

Very interested in seeing a fix for this as well (we are using pnpm-lock.yaml)

valpinkman avatar Jul 04 '23 13:07 valpinkman

Facing the same issue with pnpm-lock.yaml.

edouardr avatar Jul 05 '23 05:07 edouardr

any update on this ?

valpinkman avatar Oct 09 '23 13:10 valpinkman

Would the team accept a PR for this? I have no idea if it's something I could manage, but it might be interesting to take a stab at it.

dallonf avatar Oct 26 '23 17:10 dallonf

Update: I attempted to investigate this, but I couldn't actually reproduce the issue specifically! I wonder if it was fixed at some point. We're on v1.10.16.

dallonf avatar Nov 22 '23 17:11 dallonf

Still hapenning here, turbo 1.10.16 and pnpm 8.10.2

valpinkman avatar Nov 27 '23 14:11 valpinkman

Just to let you know this is also happening in the last version 1.11.2 (with pnpm 8.7.5)

nicolaspayot avatar Jan 03 '24 09:01 nicolaspayot

Works in 1.13.1-canary.3(with pnpm 7.33.6) Wonder if it's because using a shared workspace lock file.

Noticed when doing a package update (or adding some package) that a lot of other packages use I can see pnpm resolve and dedupe in other dependencies. So if I add something that affects the downstream of packages it'll mark as affect. So it is technically correct to mark as affected (just hard to realize sometimes).

If I add a package to a monorepo that I know will not affect any other libs (I used left-pad), filter works as expected and only marks the package I added it in as changed (this still modifies root pnpm-lock.yaml).

I didn't validate if I can find some package that doesn't affect all but more than 1.

EWhite613 avatar Mar 27 '24 21:03 EWhite613

Hey, folks! This feature has been a part of Turborepo since 1.7. Sorry for only coming across this issue just now!

For the reports that are on this issue that are using 1.7 or later, we're suspecting that dependencies are being updated deep in the tree, resulting in cache misses, as @EWhite613 has astutely noted. I just did a few similar tests as @EWhite613 and found the expected behavior.

Much of the caching logic around dependencies is drawn off of this logic so we're pretty confident that we have this working how it should. If you're still convinced you're seeing unexpected cache misses, please open a fresh issue with a reproduction so we can take a look.

anthonyshew avatar Apr 02 '24 02:04 anthonyshew

Yes, in our investigations, we've more or less isolated the problem to pnpm unexpectedly updating sub-sub-sub-sub-etc dependencies shared by many projects, rather than an issue in Turborepo's logic.

dallonf avatar Apr 02 '24 13:04 dallonf

Yes, in our investigations, we've more or less isolated the problem to pnpm unexpectedly updating sub-sub-sub-sub-etc dependencies shared by many projects, rather than an issue in Turborepo's logic.

any way of preventing this behavior from pnpm ? It's a pain to run the whole CI because I've update one package somewhere

valpinkman avatar Apr 02 '24 16:04 valpinkman

Also suffering from this issue, with pnpm v9.0.6 and also with earlier versions on pnpm, turbo v1.13.3 Tried to upgrade to v2.0.4 but still had the same issue Most of the PRs that changes the pnpm-lock.yaml triggers the entire mono-repo tasks

tomshudev avatar Jun 19 '24 11:06 tomshudev