[Proposal] Use tgz, behave like publishing to a registry

[javier-garcia-meteologica]

I propose to make yalc simulate the behavior of a npm registry. yalc [-p npm|yarn|pnpm] publish would call npm|yarn|pnpm pack to create a tgz and then store that file in ~/.yalc/<scope>/<pkg-name>/<scope-pkg-name-version>.tgz.

When a package is installed with yalc add --tgz [-p npm|yarn|pnpm] <scope>/<pkg-name>, that package is copied to <project>/.yalc/<scope>/<pkg-name>/<scope-pkg-name>.tgz and installed in the current project using the specified package manager. If -p is not specified, read it from .yalcrc or default to npm. For instance, if -p pnpm option is used, it will call pnpm add .yalc/<scope>/<pkg-name>/<scope-pkg-name>.tgz.

The result is that <scope>/<pkg-name> will be added to package.json, it will be correctly installed in node_modules, all of its dependencies will be resolved and it will obey all the rules of your package manager.

{
  "dependencies": {
    "<scope>/<pkg-name>": "file:.yalc/<scope>/<pkg-name>/<scope-pkg-name>.tgz"
  }
}

Why?

Right now yalc publish clones the project directory in ~/.yalc/<scope>/<pkg-name>/<version> and makes some minor modifications. When a package is installed, it clones the directory to <project>/.yalc/<scope>/<pkg-name>.

If yalc link <scope>/<pkg-name> is called, the directory is symlinked to node_modules/<scope>/<pkg-name> but dependencies are not installed. In this scenario if I call npm install from <project>, it will just remove the symlink from node_modules instead of installed the dependencies of our custom package.

If yalc add <scope>/<pkg-name> is called, the dependency is added to packge.json and the directory is again symlinked in node_modules but none of its dependencies are installed. In this scenario, if I call npm install from <project>, it will throw this error:

$ npm install
npm ERR! code ENOENT
npm ERR! syscall access
npm ERR! path <project>/.yalc/<scope>/<pkg-name>/node_modules/tslib                                                                                                                  
npm ERR! errno -2                                                                                                                                                                                                   
npm ERR! enoent ENOENT: no such file or directory, access '<project>/.yalc/<scope>/<pkg-name>/node_modules/tslib'                                                                                                                                                                            
npm ERR! enoent This is related to npm not being able to find a file.                                                                                                                                               
npm ERR! enoent

This error is produced because the resolution algorithm of package managers is incompatible with this declaration. They resolve the directory .yalc/<scope>/<pkg-name>, assume it's a develpment directory and that the user should take care of installing the dependencies.

To workaround these issues I have to install manually the dependencies for each package that I've added with yalc

$ cd node_modules/<scope>/<pkg-name>
$ npm install

How do tgz packages solve this issue?

When you declare a tgz dependency, the package manager unpacks the archive and installs all of its dependencies automatically.

Advantages

All package managers must maintain compatibility at the package-format and registry levels. These are therefore the most stable segments in the publishing-consuming process. Complying with the package-format standard gives yalc interoperability and stability.

Delegating packaging and installation to package managers helps yalc achieve better integration with these tools, and therefore integrates better in users' projects. Since those responsibilities are offloaded to package managers, yalc becomes more manageable.

What about backwards compatibility?

I think the new cli can still keep backwards compatibility, even if the internals are refactored. When yalc add or yalc link are used, just unpack the tgz file in .yalc/<scope?>/<pkg-name> and continue from here.

[wclr]

In this scenario, if I call npm install from , it will throw this error:

I'm not sure why you get the error for your case. You can make repro repo or steps to reproduce, I will look into it.

UPD:

To plugin yalc'ed packages in the pnpm project I use workspaces, otherwise it won't work. I created an issue for this..

As for npm/yarn I didn't check how they treat such a case. Using workspaces (which are supported by npm and yarn) should work.

Without the use of workspaces if PM doesn't handle .yalc local dependencies right one should manually install deps in the yalced package I believe. Probably this cuold be handled by yalc itself by default, I just never used such workflow. I think you can automate this with postyalc update hook script.

I don't think that tallbar is the way to go, as with yalc we want to keep things transparent and quick.

[javier-garcia-meteologica]

I've tried my best to make the issue reproductible in this repo. It doesn't need workspaces to be triggered, just complex package depnedencies. The instructions are inside the README.md

https://github.com/javier-garcia-meteologica/yalc_install_issue

[wclr]

It doesn't need workspaces to be triggered

As I said - using workspace is a solution.

But I don't get your case, you propose to make some strange contrived setup that does look like a real-life case, at least I really don't understand what is going on there and why. If you can you may try to explain the case in the plain words.

[javier-garcia-meteologica]

The real setup in which this problem is triggered is the following:

• A frontend "project" that uses "@apollo/client".
• A monorepo of webcomponents that includes "@deps/dep1", "@deps/dep2" and "chalk2". It's managed using pnpm + workspaces.

I decided not to use a monorepo/workspaces in the reproduction to simplify the analysis. For this case, yalc is a valuable tool to make changes in some of the webcomponents and test those changes directly in a "project" during development before publishing a new version. Since there are multiple projects that use these webcomponents, each project has its own repository and shared webcomponents have their own monorepo. That's why the "project" package can't be a workspace of the webcomponents monorepo.

[wclr]

I think you should simplify the repro case to narrow down the issue. Currently, there some stuff with typescript/rollup/ect. is not related, and some strange manipulation with replacing deps file: paths to absolute, this stuff looks weird. So try to demonstrate the problem in a more simple way, so I have only run a couple of commands from terminal and get the error.

[javier-garcia-meteologica]

This time I've tried to make it easier while maintaining the issue reproductible. There script prepare.sh takes care of step 1, so you can go faster to step 2.

https://github.com/javier-garcia-meteologica/yalc_install_issue

I can't get rid of typescript/rollup/etc because the issue goes away. I suspect that when npm install a dependency from file:/directory/, it tries to execute build scripts of the dependencies. That doesn't happen when file:my-package.tgz is used.

I know it's complex, but I can't make it any simpler.

[nthurow]

I just ran into a similar issue that I believe would be solved by this approach. I recently upgraded to yarn berry, and had to change my files key in package.json because apparently Yarn berry isn't perfectly compatible with NPM in how it resolves files (see https://github.com/yarnpkg/berry/issues/1937#issuecomment-743915311). However, I was just made aware today that yalc no longer works in our project, because yalc resolves files the way that NPM does.

So I'm kind of stuck. But I think if Yalc used the underlying "npm/yarn pack" command, this wouldn't be an issue.

[wclr]

So I'm kind of stuck. But I think if Yalc used the underlying "npm/yarn pack" command, this wouldn't be an issue.

What it should produce using this command?

[nthurow]

If I'm understanding @javier-garcia-meteologica's proposal, it would produce a .tgz package (which is just the result of running yarn pack or npm pack) and store that in the .yalc repository.

[cpakken]

I think the option to add a tgz package in the .yalc directory when yalc add is run would be a very useful feature. It would provide consistency in how different package managers handle the yalced dependency.

To my understanding, the idea is when a package is yalc publish it will copy the package.json configuration to a global store. In addition, it should also convert these files into tgz as well. When the user yalc add --tgz, then it will clone the stored tgz into the .yalc directory instead.

This way pnpm and yarn 2 (berry) will also automatically install their dependencies via their own methods consistently.

[wclr]

I believe this could be added but with yalc pusblish --tgz probably. So that one on the publishing side chooses the way it should work, and yalc add will handle what is in store for this package. I'm not sure if we need to allow different ways of publishing for the same package at the same time.

[nthurow]

Would be really great to see this implemented, as yalc has been an extremely useful tool for us. Please let me know if I can help in any way.

[wclr]

You could make a PR, if you have a desire. It shouldn't be difficult.

[nthurow]

You could make a PR, if you have a desire. It shouldn't be difficult.

👍 I'll give it a try.

[wclr]

if possible prefer adding new modules (files) to changing existing ones.

[metasean]

I'm curious where this currently stands.

I also have another possible use case - when the published package needs to be installed globally. I can easily yalc publish and yalc add -g my-package, across MacOS, Debian, & Windows, but then I have to find where that global package has been installed _for the particular _OS and do the npm install/yarn from that global install location. It's definitely doable, but it's not obvious —it's definitely convoluted. I think @javier-garcia-meteologica's suggestion would remove the non-obvious/convoluted aspects.

That said, overall yalc has definitely made my life easier!