Support Yarn-pnp, esy, and npm-link

[jordwalke]

The following workflows and package managers don't work when tools build into node_modules directories, or rely on dependencies existing in node_modules.

Because of this, developers building native apps with esy cannot use bs-platform at all. They have very frequently requested the ability to use esy to build bs apps, but there's not much esy can do at this point, and the solution to esy is the same as the solution to yarn pnp and npm link, so might as well solve all three at once.

1. Yarn Plug'n'play. (Stores dependencies in global cache).
2. esy used to manage JS dependencies (uses yarn plugin'n'play standard).
3. Npm link workflows. Two packages can link to the same dependency, with different build flags so putting build artifacts into that dependency's root will have artifacts conflict with each other).

Here's what I believe needs to happen to make bs-platform work with all three of these. The solutions are largely the same for all use cases:

Stop Building Into Dependencies' node_modules

Solves Problems with: Yarn pnp, Esy pnp.

The idea here is that all libraries should be built into the top level project's root directory. Separate Tracking Issue

Don't assume dependencies even exist in node_modules.

Solves Problems with: Yarn pnp, Esy pnp.

Since esy and yarn both never touch the disk if it can read the dependencies from an immutable cache, we can't assume the packages are located at node_modules. Instead yarn pnp generates a .pnp.js and esy generates a ./_esy/default/pnp.js file. That tells you the real locations of dependencies in the immutable cache. Esy uses the exact same pnp standard, but we can make some changes if it makes it easier for you.

Optimization: Yarn pnp requires starting up a JS VM to determine the location of dependencies. I believe we can make esy's pnp.js file carefully constructed so that bsb could just read the locations directly from the text file by parsing simple strings instead of starting a JS VM. It would also work if you started it up with a JS VM but we can overlay an additional protocol on top of yarn pnp for the sake of additional performance.

Avoid postinstall for bs-platform's installation.

Solves Problems with: Yarn pnp. Esy and npm can still tolerate a postinstall step for bs-platform, but yarn pnp cannot. It's better to avoid it if possible so that even yarn pnp can work well. If not, then at least esy pnp and npm link workflows would still work.

Model compiler explicitly:

This one is only relevant for esy. But if there is an esy.json file alongside package.json for bs-platform, which expresses the dependency on the ocaml compiler it uses, and its version, then esy could be used to install native ocaml ppx's for BuckleScript projects.

[texastoland]

Keeping notes as I look. AFAIK the main change needs to happen in Bsb_pkg. Every other reference to node_modules is either a template or a test. Also relevant is @bsansouci's stale (what happened?) #2514.

[texastoland]

I believe we can make esy's pnp.js file carefully constructed so that bsb could just read the locations directly from the text file by parsing simple strings instead of starting a JS VM.

The manifest file appears resolved by https://github.com/yarnpkg/yarn/issues/6388#issuecomment-471498188.

[huy-nguyen]

Is there anything a newcomer like me can do to help this gain some traction? Where should I look to get started?

[BlueHotDog]

Hi, pushing this up. Yarn Berry, the next version of Yarn is moving fast and gaining traction. TS and all major Js tooling support this already. can we push this higher?

[chriswilty]

It's gone awfully quiet in here.... Has this been dropped, or just greatly deprioritised?

I didn't even know it was a problem until I started a project with Yarn 2, and wondered why nothing was working, so at least add a warning to the Rescript docs.

[fhammerschmidt]

@chriswilty have you seen https://github.com/reason-seoul/yarn-plugin-rescript/ ?

I think it should be mentioned in the docs somewhere, it is still completely new though.

[cometkim]

I agree to the assumption that node_modules exists is wrong, but this is too pervasive.

So even the Yarn PnP's approach is better than node_modules, in reality there are many problems with adaptation and that why node_modules linker exist.

What I did in yarn-plugin-rescript was to provide a compatibility layer. I'm thinking if it's possible in other places in a similar way.

[ruittenb]

Is this issue still on the map?