Update knit proposal

[devongovett]

Based on my experiments on https://github.com/yarnpkg/yarn/pull/3753, I'm updating the knit RFC with a few additional sections as requested by @bestander:

1. Added yarn install --knit option to automatically link dependencies when installing.
2. Section comparing several other yarn features for linking deps.
3. Mentioned requirement of --preserve-symlinks option in node for this to work correctly.

[bestander]

Awesome! Thanks, @devongovett, for stepping up

[bestander]

@devongovett, what do you think of making knit just a for of link and not a separate command? E.g. all link:/ dependencies would be knit instead of symlinked to the source if yarn install/add/upgrade is called with --knit-link flag?

That might reduce the terms overload on users.

[devongovett]

@bestander so you mean the following?

• yarn link --knit - same as yarn knit in RFC
• yarn add dep --knit - does what yarn knit dep did in the RFC
• yarn install --knit - links all dependencies that have been setup with yarn link --knit

[bestander]

• yarn link --knit - same as yarn knit in RFC

Yes

• yarn add dep --knit - does what yarn knit dep did in the RFC

Only if dep is like dep@link:../path/to/dep

• yarn install --knit - links all dependencies that have been setup with yarn link --knit

And also all dependencies with link:/ and file:/ (when used as link:/) specifier.

[devongovett]

1. is fine
2. doesn't make sense to me because you'd need to know the path to the global yarn registry of knit dependencies to specify the link path. I'm on board with yarn add dep --knit means install dep from the knit registry though.
3. So it would treat file:// deps as link://? I suppose so, but this seems separate from knit - i.e. it has nothing to do with deps in the knit registry.

[devongovett]

cc @ide who wrote the original knit proposal

[bestander]

What I tried to say that knit is a mode of link feature, the implementation could probably piggy-back ride on the link with a conditional action at the end.

And we can do linking 3 different ways:

• yarn link command
• yarn install/add/upgrade with package.json that contains link: dependencies
• yarn install/add/upgrade with package.json that contains file: dependencies but called with yarn-link-file-dependencies true in yarnrc, this basically makes them behave as link: dependencies

So knitting should be compatible with all of them

[devongovett]

Ah I see, sorry for the misunderstanding.

Yes, I think knit could be a mode of link:

• yarn link --knit - same as yarn knit in RFC.
• yarn link dep --knit - same as yarn link now, but also installs dependencies of dep locally.

What do you think of dropping the need for yarn link dep and just using yarn add dep --knit instead though? This would have the effect of also automatically linking any sub-dependencies of dep which might have been setup locally (with yarn link --knit) as well, since implementation of add is shared with install. This might have happened previously with yarn link because it included node_modules, so sub-dependencies of dep that were linked would also be carried over.

[bestander]

Yes, I think knit could be a mode of link:

• yarn link --knit - same as yarn knit in RFC.

👍 What do you think of giving it a more self explanatory name like --prepublish or --prepare?

• yarn link dep --knit - same as yarn link now, but also installs dependencies of dep locally.

Considering that at step one, yarn link --knit, we simulated a publishing to npm by putting files to a temp/cache folder it should create a symlink not to the folder of knitted project but to the folder where we put the build artifacts. So it would work slightly differently.

What do you think of dropping the need for yarn link dep and just using yarn add dep --knit instead though?

I am worried that it will be confusing. Would it create a symlink or would it install dep by copying files? link implies that the dependency will be installed via a link from a local place.

[devongovett]

Sorry for the delay in reply.

Yes, I think the knit name isn't very descriptive. In my POC pull (https://github.com/yarnpkg/yarn/pull/3753), I called it yarn link --deep because it makes links to each file rather than the directory. Maybe that makes sense? Ideally this would be the default mode for yarn link, but I suppose we can't change compatibility with npm link. Maybe a config option in yarnrc to make it the default?

yarn add dep --knit is a bit confusing, but what about yarn add dep --link? Then it's clear it's linking. It also seems clearer than yarn link dep since it is both linking and installing dependencies, whereas yarn link dep is only linking.

So my proposal would be:

• yarn link --deep - links each file in the cwd to the global link registry
• yarn add dep --link - links dep to the global link registry and installs its dependencies
• yarn install --link - installs all dependencies and auto-links ones that are in the global link registry

No new commands, only two new options.

[ide]

That naming proposal mostly makes sense to me. I prefer a flag other than --deep like --pack (as in yarn pack) or --publish since part of the RFC's intent is to mimic what happens when you publish a package. If your package has src and build directories and your package.json specifies "files": ["build"], then only the build directory should get symlinked (and shallowly, not the files within it).

I like the link:// URLs too, really useful for internal projects.

[devongovett]

I think yarn link --pack is probably the best option so far. --publish might be confused with actual publishing I guess.

[devongovett]

Another open question is what put in the lock file. I would say that installing with the --link option would not affect the lock file: it would remain as if you had installed from the normal package registry. Otherwise we might run the risk of committing changes to a lock file that don't work in a real environment. The only case where this doesn't work is if the package isn't published to the registry yet. In that case I guess it would work as link:// urls do now, and you would have to update it manually when the package is published. Thoughts?

[bestander]

link --pack sounds good

[bestander]

As for yarn.lock - good question. What would be the result of this operation:

yarn link project-b --pack
yarn add left-pad

Would the second command overwrite what is inside node_modules? I think lockfile should indicate something to conform with the promise that the same yarn.lock produces the same node_modules

[devongovett]

yeah its a bit weird if you commit your yarn.lock file though, since you may end up with local-only artifacts being committed.

[devongovett]

Updated the RFC with the new naming as discussed above, and renamed the file. Sorry if it makes it harder to see the diff. :/

[devongovett]

@bestander @ide thoughts on the update? Would like to get started on an implementation if you think this is good.

[bestander]

@devongovett, I think we are almost there, thank you for your work on this feature, it should be really useful. I've added a few suggestions above

[bestander]

Looks like this PR got stuck :( I'll be happy to iterate more on the details.

[devongovett]

@bestander sorry about that. I got really busy, and a bit confused about what the next steps are for this proposal. However, we really are in desperate need of this feature, so I am happy to work on it if we can come to a well-defined agreement about what we need.

Here is my proposal to move things forward:

1. Split this proposal out into a separate document, leaving knit as it is. We can update the current doc to be much more clear about how the feature should work on some of the finer points discussed in the comments.
2. yarn link --pack (or some other name) should run prepublish scripts, and link all files that would be published (following .npmignore, excluding node_modules, etc.) into a global directory, just as yarn link works currently, but linking individual files instead of the project directory.
3. yarn install --link installs all dependencies, and for packages that are in the global link directory, those packages are linked.

The global directory is needed so that yarn install --link knows which projects are available to link. It also enables changes to be made within a project and automatically propagate those changes to all places that use it.

Please let me know if there are other things that we should cover.

[bestander]

Hey, @devongovett, no worries, I can totally understand. Same here, I got busy with other projects and could not be very responsive recently.

I wish Yarn would have a plugin API so that people could code their needs to get unblocked.

On your proposal.

1. +1 let's make the minimal possible change you need to get unblocked. It is way easier to get a smaller feature in. And +1 on updating the current doc with a few comments guiding the future contributors.

2. Ok, we can discuss the flag name and the folder layout in the new RFC.

3. Do we need this flag? It seems more consistent if yarn link pkg-a links a package that was declared with yarn link command before.

[fenduru]

How will yarn knit work when knitting a package that is in a monorepo? For instance imagine the following directory structure

/
  my-mono-repo/
    packages/
      foo
      bar
  baz/

Where bar has a dependency on foo, and from baz you run yarn knit ../my-mono-repo/packages/bar. Without special handling for workspaces I'm not sure that the current knit proposal would properly pull in the local copy of bar or not.

[arcanis]

This rfc is pretty old, and predates our workspace feature which is a much safer alternative to yarn link. I don't think we'll implement yarn knit.

[SamB]

@arcanis: workspaces may be safer, but they seem to be fairly explicitly intended for planned usage, not ad-hoc "oh man, there's an issue in one of our dependencies, let me go make a PR for that, testing it in our thing so I can easily iterate on fixing our actual problem" use cases.

So, features that would actually help with those ad-hoc use cases are still desirable. Since yarn link exists and can be used for this, it is the baseline against which other approaches would be measured (with presumably a glance or two at npm link), and the advantages weighed against (a) the maintenance burden, (b) the expected cognitive load on users trying to decide what to do, and (c) is it likely that someone will soon discover a much more elegant approach?

Note: I have no position on whether any given yarn knit proposal would in fact improve things.

[Hmm. I'm really not good at writing short comments, am I?]