Yarn2 support

[armarti]

Hey there, this is a neat project and could hopefully someday do away with needing ts-node for everything.

I want to open this as more of a TO-DO for when I or someone else finds time. Right now I'm using yarn2, which uses a virtual filesystem and doesn't have a node_modules folder. All packages are instead stored in .zip files and extracted when needed.

A few takeaways from my hour trying to get this to work with yarn2.0.0-rc.29 and node 13.9:

• It looks like this package depends on there being a node_modules folder, so that'd need to be dealt with

• Yarn2's virtual filesystem is accessible via a few different routes but the simplest is by starting node as node --require ./.pnp.js from the project root. Delving into exactly what happens when .pnp.js is hooked like this is a big TODO, but I believe it monkeypatches the fs module to automatically unzip packages whenever they're imported / required.

  The catch22 with using yarn2 + this neat package is:

  • When package.json has "type": "module", require is not defined at startup. Since the CLI flag --require ... seems to literally use require(...), this fails with an error.
  • Without --require ./.pnp.js, none of the workspace's packages are available, meaning --loader=@k-foss/ts-esnode results in Cannot find package '@k-foss/ts-esnode ...

So maybe sometime later this month I'll have some bandwidth to tackle this. But compliments again for finding a way to leverage these new node features to potentially make typescript less painful.

[EntraptaJ]

Sorry I missed this earlier. I think I know why this is failing and I'm not sure how it could be resolved without finding how to remove my hack for supporting commonJS modules, because without my dynamic module type all imported node modules not ESNext format give a XYZ does not export ABC, if you had something like this:

import { ABC } from 'XYZ'

So to solve that and allow for compat with older modules or TypeScript projects with ESModule compat enabled I have to load all node_modules as dynamic modules with a dynamicinstanciate hook that manually imports with a createRequire and then returns the values on the object and spreads out anything withing imported.default. It's super hacky but it was the only way to achieve my goal of drop in support in most projects without any refactoring.

Here is how I load all "node_modules"

/**
 * This dynamically imports the `node_modules` module and creates a dynamic module with all the same exports.
 * @param url fileURL given by Node.JS
 */
export async function dynamicInstantiate(url: string) {
  // Create a Node.JS Require using the `node_modules` folder as the base URL.
  const require = createRequire(
    `${url.split('/node_modules/')[0].replace('file://', '')}/node_modules/`,
  );

  // Import the module file path
  let dynModule = require(url.replace(/.*\/node_modules\//, ''));

  /**
   * This is needed to allow for default exports in CommonJS modules.
   */
  if (dynModule.default)
    dynModule = {
      ...dynModule.default,
      ...dynModule,
    };

  const linkKeys = Object.keys(dynModule);

  return {
    exports: [...linkKeys, 'default'],
    execute: (module: any) => {
      module.default.set(dynModule);
      // For all elements in the import set the module's key.
      for (const linkKey of linkKeys) module[linkKey].set(dynModule[linkKey]);
    },
  };
}

[EntraptaJ]

That hack is the only part of this codebase I hate/want to get rid of. If anyone has any solutions or alternative ways of supporting destructing of legacy node_modules please submit a PR or let me know and I can attempt it.

[EntraptaJ]

I will attempt to refactor the file and module loader functions this weekend.

[EntraptaJ]

@armarti Can you test to see if the issue is still present with the changes made in #28 I changed how I require in Common.JS modules, which could possibly allow the Yarn2 hack to redirect require to work, although I really don't want to setup Yarn myself.

[armarti]

Ha just seeing this again, impressive progress you've made. With luck I'll get back to the project I was working on in the next week or two and I could try this out.

[EntraptaJ]

Yeah, when I started this project it was meant as a proof of concept to be integrated into TS-Node now I've morphed it into its own ecosystem.

[EntraptaJ]

Wait, I wonder if my new getSource based CJS module compat/(TOTALLY NOT A HACK) makes Yarn2 Work now...

[EntraptaJ]

Doesn't work magically but I do think that this will be easier to be done

[EntraptaJ]

The core of this issue is that Yarn2 doesn't support NodeJS loader hooks, because their hack to allow for the "compressed" modules assumes that ESModules don't exist in Node.JS. This can for sure be done. I'm not sure as to how complex this will be. I myself don't use Yarn2. And as much as I'd love to solve this myself, unless I decide to understand the internals of Yarn2 I don't think I can do this myself.

[EntraptaJ]

One way I could see this working is if TS-ESNode is loaded from disk instead of attempting to load the installed version from Yarn

[EntraptaJ]

Just thought of a potential workaround, using a bin script that loads TS-ESNode. Turns out that even that won't work because the virtual filesystem appears to not include the package.json file that includes the "type": "module JSON