Yarn --production tries to resolve devDeps as well

[Diokuz]

Do you want to request a feature or report a bug? This is a bug, or unexpected behaviour.

What is the current behavior?

1. Have a package.json which have devDependencies and one of them are private
2. Run yarn --production
3. Yarn tries to resolve all packages and fails, because have no access to private packages

To fix that I need additional workaround with authToken (the problem arises in docker build process).

What is the expected behavior? Yarn should only resolve production packages when --production flag exposed.

Please mention your node.js, yarn and operating system version.

node -v
v8.1.0
yarn --version
0.24.6

[rally25rs]

This happens because Yarn needs to consider the devDependencies so that it can produce a deterministic build between prod and dev builds (packages will be hoisted to the same locations).

[Diokuz]

Not sure I got it... lets consider two situations:

1. (rm -rf ./node_modules) We have package.json with deps and devDeps, and we run yarn --production
2. (rm -rf ./node_modules) We remove all devDeps from that package.json and run yarn

Will there be the difference in node_modules? If yes, why?

[Diokuz]

Actually, I think this is related to my another problem https://github.com/yarnpkg/yarn/issues/1379

[rally25rs]

Possibly. Consider something like:

dependencies {
  [email protected]
}
devDependencies {
  [email protected]
}

And A also has a dependency:

dependencies {
  [email protected]
}

---

With devDependencies, the dependency tree is:

[email protected]
|- [email protected]
[email protected]

so A@1 can not longer be hoisted to top level, resulting in:

node_modules/
|- A/ (1.0.0)
|  |- node_modules/
|     |- B/ (1.0.0)
|- B/ (2.0.0)

---

Without devDependencies, the dependency tree is:

[email protected]
|- [email protected]

But B can be hoisted to the top level, so the result is:

node_modules/
|- A/ (1.0.0)
|- B/ (1.0.0)

(This is non-deterministic because B@1 could end up in 2 different paths; either in node_modules/B or node_modules/A/node_modules/B depending on the type of install)

---

What Yarn wants to do for a deterministic build is to make sure that [email protected] is installed to the same location every time, so a --production build should notice that [email protected] will be hoisted to top level if the devDependencies were there, so the resulting install should really be:

node_modules/
|- A/ (1.0.0)
   |- node_modules/
      |- B/ (1.0.0)

---

I hope that makes sense :) Basically all the devDependencies are built into the dependency tree when figuring out where to hoist packages to, and calculating that dependency tree needs the metadata for all those packages.

[rally25rs]

@Diokuz I haven't tried it myself, but you might see if NPM5 has the same issue. If I understand the NPM5 lockfile correctly, I think it saves the resolved / hoisted position for each package in their lockfile (Yarn does not), so it might not need to query the metadata for your private repos. As much as I hate to steer anyone toward NPM, this might be a case where NPM5 works better. It's worth a test anyway!

You might also be able to solve this using the Yarn offline mirror, since the metadata and package would be saved there and shouldn't have to query the actual server. However that might have other security concerns for you, since a copy of your private repo code would be in that mirror directory.

[Diokuz]

@rally25rs thank you so much for explanation!

So, I think I get my answer. The only question now: will that be done in near future? I mean, saving positions in lock file.

[arcanis]

It's not only a question of position. Let's say you have this package.json:

{
    "dependencies": {
        "a": "^1.0.0"
    },
    "devDependencies": {
        "b": "^1.0.0"
    }
}

And that the following are true:

• [email protected] and [email protected] both exist
• [email protected] exists and depends on [email protected]

If we don't resolve devDependencies, then in prod we'll have the following:

/node_modules/
/node_modules/a (@1.1.0)

And in dev, because of the deduplication optimization, we'll have the following:

/node_modules/
/node_modules/a (@1.0.0)
/node_modules/b (@1.0.0)

Note that we end up using a different version of a in prod than in dev - that might not be good.

[arcanis]

Now that I think about this, maybe in practice it could work, since the lockfile would still pin the version - at least assuming that this lockfile has been generated in dev mode.

Still, I'm not convinced this is an issue worth fixing right now. I feel like this is something that could probably be better solved through your build environment.

[Diokuz]

Well, we have solved our problem via NPM_TOKEN env var. But if npm is better here, why dont you want to make yarn even better in near (or far) future?)

Anyway, thanks for answers)

[kentor]

@arcanis We don't control the build environment. For example I am using Netlify which does not have access to private dependencies through github (not npm private registry, so I can't do something like NPM_TOKEN). I thought it would be ok to move the private deps to devDependencies but yarn in production mode still tries to resolve it. The lock file should be enough to determine the positions of the packages.

Edit: Just tested npm5. It does not need to resolve devDependencies

[saikojosh]

I love Yarn but this has repeatedly been raised as an issue since last year and is always closed. Surely this is a common use case to have access to different packages in production and development?

My own situation is that I'm using a private Git dependency in development and in CI/production I am using a private module hosted on NPM. I don't want to give my CI system or production systems access to GitHub - Yarn should not require access to that dependency. If NPM can figure out a solution that works, so can Yarn.

For now I am going to have to switch back to NPM.

[adrienharnay]

Hey, bumping into this issue at well.

yarn --version
0.27.5

How to reproduce:

• Create an empty folder, cd into it
• Create a package.json:

{
  "devDependencies": {
    "doesnt-exist": "file:./nope"
  }
}

• Run: yarn --production

npm i --production does work.

[BYK]

Have any of you tried using --prefer-offline or --offline? That should work IMO.

[adrienharnay]

Just tried, doesn't seem to work.

[olingern]

@BYK Opened a PR for this fix: #4210

[thedaniel]

That PR has been closed without merging, any idea when a new one might make it onto the schedule?

[olingern]

The issue is slightly complex. I believe the solution could be attaching metadata of sorts to the lockfile, but I defer to @arcanis @BYK

[BYK]

The thing is, we still need those package.json files to have a consistent image of the resolved file tree, even if those dependencies won't be installed.

An alternative would be to use the offline mirror feature that would include the dev dependency packages or, their stripped-down versions which only contains the package.json files.

@olingern's suggestion also sounds good but I don't know what the implications for that are. @kaylieEB, @arcanis, @bestander - any thoughts?

[rally25rs]

I would imagine we could add a "node_modules path where the packages is hoisted to" in the metadata. I think that would eliminate the need to re-query the metadata to build the hoisted dep tree. To me the bigger issue is compatibility. Maybe this becomes something for a yarn v2 roadmap?

[bestander]

I think there are a lot of ways to go around this issue: linking, workspaces, fake offline mirror + tweaking yarn.lock. And this issue is IMHO not common enough to justify some significant changes to package resolver which is the core of Yarn.

At this stage a better solution is to document the easiest way to go around this limitation and send a PR to our blog https://github.com/yarnpkg/website/tree/master/_posts

[adrienharnay]

I have found several people with this problem, which doesn't occur on NPM. Sometimes you have a local dependency which you don't want to include in production, and you have to use NPM on the production env (docker or such), which can lead to problems, caused by the differences between Yarn and NPM. Could you please ELI5 why this works on NPM but not on Yarn? Thanks!

[bestander]

Yarn is designed to produce the same results on all computers, it verifies that all dependencies are present and fold/deduplicate the same way. E.g.

A -> B -> C@^1.0.1
  -> (devDependency) D -> [email protected]

Yarn will try to install A, B, C, D flat and resolve versions ^1.0.1 and 1.3.0 to a single version. If production build ignored D branch it might have fetched a different version of C. Before Yarn people ended up having different non development dependencies (A, B, C) in dev and production mode.

That is why Yarn requires for all the four to be present at resolution time. There are ways around this that would satisfy your case without making changes to Yarn, e.g. use a file: or link: specifier for the local dependency in dev mode and provide a fake one in your production environment.

[adrienharnay]

Thanks for the explanation! Wouldn't resolving devDependencies last (quite a naive approach) be a solution? This way, no dependencies would depend on devDependencies.

[bestander]

The order does not matter much, Yarn still tries to de-duplicate deep dependencies (C in the example above) and move them to the root of node_modules.

[adrienharnay]

I read this post a few weeks ago, which states that "npm 5 has stronger guarantees across versions and has a stronger deterministic lockfile". Then how can it be than it works without resolving devDependencies? Core implementation? (I'm trying to fully understand why the problem can't be "fixed" here. I understand there are workarounds which can be used to make Yarn do the job)

[bestander]

@Zephir77167, this is a great discussion to have but this issue is not a place to have it. Feel free to jump into the chat https://discordapp.com/invite/yarnpkg and discuss it with the community.

[adrienharnay]

Alright, thanks a lot for taking the time to answer!

[ahmadnassri]

still seeing this with v1.3.2

[heisian]

OK, but why include devDependencies in the --production dependency tree to begin with?? Isn't the whole point not to include/resolve ANY of those devDependencies and their dependencies? I don't care about hoisting devDependencies' dependencies.

Is your reasoning as such: "Since devDependencies' dependencies are hoisted in development, we want to ensure consistent behaviour in a production environment by also hoisting those same dependencies, even if their parent modules aren't installed/used." ???

[rally25rs]

@heisian Yeah, your statement is basically correct. It's to produce a deterministic node_modules hoisting tree, which is one of Yarn's key features. I wasn't part of the original team that created Yarn and implemented all that, so I'm not sure if having to re-resolve devDeps was an actual design decision, or just a side-effect of the implementation. There has been talk of putting the hoisting tree location in yarn.lock which would mean Yarn would no longer have to resolve those devDeps, but that would change the structure of the lock file, so would likely warrant a major-version change (yarn v2) and hasn't been worked on yet as far as I know.