Add an ability to import and run JS modules in server-side components

[tipiirai]

This should be possible:

<script>
  import { hello } from './hello.js´
</script>

<div>
  { hello() }  
</div>

Currently Nue uses a new Function() call to execute server side scripts, which does not allow import statements an is limited in many ways. A better way is to use dynamic imports.

Unfortunately Bun does not yet support imports via plain string so we need to figure out something clever. Maybe use Bun.build or something. I don't know.

What is the best way to implement this? I know this is a rather heavy-duty question and requires deep understanding of both Bun/Node and Nue, but yeah... I'll ask anyway.

[hichem-dahi]

Can't we use eval function (although it may not be secure) or it's alternatives?

[tipiirai]

I don't think you can import modules with eval.

[tipiirai]

ie. I'm open to any option that works. Let's tinker with the possible security issues / other concerns later.

[hichem-dahi]

In the example you've shared, we have

...
    const text = await (await fetch(url)).text();
    const dataURL = `data:text/javascript;base64,${btoa(text)}`;
    const { Buffer } = await import(dataURL);
...

Can't we replace it with

...
    const text = await (await fetch(url)).text();
    eval(test);
...

[tipiirai]

I'm not sure how eval can mimic sentences like import { hello } from './hello.js´. Worth a try. Getting the hello variable might be tricky.

[hichem-dahi]

Why would it be tricky? Wouldn't eval parse it normally?

[kon-pas]

@tipiirai Did you work this out?

I came up with a fairly simple, but very primitive solution to transpile the source by hand. We could look up all import statements and transform them to dynamic imports, which are allowed in non-module environments, like eval and Function.

Here is my sample implementation. It uses a regex for the pattern lookup, so it's very error-prone. It works, but I'm not convinced anyone should use this:

const BunTranspiler = new Bun.Transpiler({
  loader: 'js',
  // minifyWhitespace: true, // TEMP: Commented out for debugging reasons
})

function transpileImports(source) {
  const imports = BunTranspiler
    .scanImports(source)
    .filter(imp => imp.kind == 'import-statement')
    .map(imp => imp.path)

  // Slightly modified version of a regex made by Antón Kryukov Chinaev @antonkc
  // https://github.com/antonkc/MOR/blob/main/matchJsImports.md
  const regex = /(?<=(?:[\s\n;])|^)(?:import[\s\n]*)((?:(?:[_\$\w][_\$\w0-9]*)(?:[\s\n]+(?:as[\s\n]+(?:[_\$\w][_\$\w0-9]*)))?(?=(?:[\n\s]*,[\n\s]*[\{\*])|(?:[\n\s]+from)))?)[\s\n,]*((?:\*[\n\s]*(?:as[\s\n]+(?:[_\$\w][_\$\w0-9]*))(?=[\n\s]+from))?)[\s\n,]*((?:\{[n\s]*(?:(?:[_\$\w][_\$\w0-9]*)(?:[\s\n]+(?:as[\s\n]+(?:[_\$\w][_\$\w0-9]*)))?[\s\n]*,?[\s\n]*)*\}(?=[\n\s]*from))?)(?:[\s\n]*((?:from)?))[\s\n]*(?:["']([^"']*)(["']))[\s\n]*?;?/gm

  return BunTranspiler
    .transformSync(source)  // Strips comments, minifies whitespaces, etc.
    .replaceAll(regex, (input, defaultImport, wildcardImport, namedImports, from, moduleName, quote) => {

      // Just to be more confident, that the import statement is indeed an import statement,
      // though it's incorrect in some rare cases, but more on that later on
      if (!imports.includes(moduleName)) return input

      // TODO: Resolve the path to run the script in the right context
      const path = quote + moduleName + quote

      if (wildcardImport)
        if (defaultImport)
          return `const ${wildcardImport.split(' ').at(-1)} = await import(${path}), ${defaultImport} = ${wildcardImport.split(' ').at(-1)}.default;`
        else return `const ${wildcardImport.split(' ').at(-1)} = await import(${path});`
      else if (namedImports)
        if (defaultImport)
          return `const ${namedImports.replaceAll(' as ', ': ').slice(0, -1)}, default: ${defaultImport} } = await import(${path});`
        else return `const ${namedImports.replaceAll(' as ', ': ')} = await import(${path});`
      else if (defaultImport) return `const { default: ${defaultImport} } = await import(${path});`
      return `await import (${path});`
    })
}

Given the source:

import defaultImport1 from "./module";
import * as wildcardImport1 from "./module";
import { namedImport1 } from "./module";
import { namedImport1 as alias1 } from "./module";
import { default as alias2 } from "./module";
import { namedImport2, namedImport3 } from "./module";
import defaultImport2, * as wildcardImport2 from "./module";
import { namedImport4, namedImport1 as alias3, /* ... */ } from "./module";
import defaultImport3, { namedImport5, /* ... */ } from "./module";
import "./module";

It transpiles to:

const { default: defaultImport1 } = await import("./module");
const wildcardImport1 = await import("./module");
const {namedImport1} = await import("./module");
const {namedImport1: alias1} = await import("./module");
const {default: alias2} = await import("./module");
const {namedImport2, namedImport3} = await import("./module");
const wildcardImport2 = await import("./module"), defaultImport2 = wildcardImport2.default;
const {namedImport4, namedImport1: alias3} = await import("./module");
const {namedImport5, default: defaultImport3 } = await import("./module");
await import ("./module");

And in action:

// module.js
export default 42
export const namedImport1 = 42
export const namedImport2 = 42
export const namedImport3 = 42
export const namedImport4 = 42
export const namedImport5 = 42

const source = `
import defaultImport1 from "./module";
import * as wildcardImport1 from "./module";
import { namedImport1 } from "./module";
import { namedImport1 as alias1 } from "./module";
import { default as alias2 } from "./module";
import { namedImport2, namedImport3 } from "./module";
import defaultImport2, * as wildcardImport2 from "./module";
import { namedImport4, namedImport1 as alias3, /* ... */ } from "./module";
import defaultImport3, { namedImport5, /* ... */ } from "./module";
import "./module";

console.log({
  alias1,
  alias2,
  alias3,
  namedImport1,
  namedImport2,
  namedImport3,
  namedImport4,
  namedImport5,
  defaultImport1,
  defaultImport2,
  defaultImport3,
  wildcardImport1,
  wildcardImport2,
})
`

const transpiled = transpileImports(source)
const AsyncFunction = async function () {}.constructor
AsyncFunction(transpiled)() /* =>
{
  alias1: 42,
  alias2: 42,
  alias3: 42,
  namedImport1: 42,
  namedImport2: 42,
  namedImport3: 42,
  namedImport4: 42,
  namedImport5: 42,
  defaultImport1: 42,
  defaultImport2: 42,
  defaultImport3: 42,
  wildcardImport1: Module {
    default: 42,
    namedImport1: 42,
    namedImport2: 42,
    namedImport3: 42,
    namedImport4: 42,
    namedImport5: 42,
  },
  wildcardImport2: Module {
    default: 42,
    namedImport1: 42,
    namedImport2: 42,
    namedImport3: 42,
    namedImport4: 42,
    namedImport5: 42,
  },
}
*/

Issues that I've already found:

• Regex doesn't match type imports

• Regex doesn't match string literal named imports: import { "string literal" as alias } from "module"

• Regex matches all import statement syntax instances despite its context, for example in strings. We verify that with Bun's scanImports(), but if the source code contains both a real import statement, and a fake one, they both get transpiled:

  "import 'module'";
  import 'module';
  

  Transpiles to:

  "await import ('module')";
  await import ('module');
  

[Hi-Alan]

This is a highly desirable feature, and I'm looking forward to it. I think there are 3-level component life time:

• compile-time
• server side at runtime
• client side at runtime