Module Namespacing Support for Packages

[krpeacock]

Summary

When installing a package that uses module namespacing (export * as {moduleName} from './{moduleName}), Expo's default configuration throws an error when compiling for web and iOS. This is an important tool for structuring TypeScript code, so I'd expect it to work out of the box.

Managed or bare workflow? If you have ios/ or android/ directories in your project, the answer is bare!

bare

What platform(s) does this occur on?

iOS, Web

SDK Version (managed workflow only)

4.5.2

Environment

MacOS, Node v14.15.5.

Reproducible demo or steps to reproduce from a blank project

https://snack.expo.io/kSlfJiSnR

Steps to reproduce:

• npm install @dfinity/agent
• Somewhere in the code - import { HttpAgent } from "@dfinity/agent";

[byCedric]

Hi @krpeacock! Thanks for reporting this issue. There are a few things going on, I'll list them.

1. The exported module is named HttpAgent, not HTTPAgent (as listed in Snack). I think this might just be a typo because the steps to reproduce are written correctly. But still wanted to mention it.

2. Snack doesn't support the newer "harmony" modules (esm or cjs). It only supports the older ES5/6 module syntax. This is something we have to fix, but I don't have an ETA for this. React Native and Expo supports this syntax, and it's parsed properly when using it in a local project.

3. The @dfinity/agent package requires modules that are not listed as dependencies. This is something dfinity has to fix, or you have to manually add it to your packages. The code requires both @dfinity/candid and @dfinity/principal.

4. React Native (and thus Expo) doesn't have TextEncoder. This is a limitation of the React Native JS environment. To make this work, you have to use a polyfill (e.g. like this). This is also something dfinity probably should fix on their end to improve support for React Native and Expo.

After applying all of this, it looks like the module is loaded properly. I haven't tested the actual module itself though, I'm not sure how to use it 😅

Hope this helps!

[krpeacock]

Thanks for the thorough investigation! You're totally correct on 1 and 3 - I stopped adding to the example once I could reproduce the error I was aware of.

4 is another known issue, but we'll avoid shipping a TextEncoder polyfill due to its wide browser support and that there are good options available on npm that a RN dev could include.

As for 2, I think that's the heart of the issue, and I don't think it's a Snack bug. I first reproduced the issue on my dev environment using the Expo CLI. Here's a more complete example to test locally: https://github.com/krpeacock/dfinity-expo-proof-of-concept

[byCedric]

Thanks for that! I can't seem to get the repo working, it's missing this file:

./.dfx/local/canisters/counter/counter.js

Is that correct? I don't see a .dfx folder in the repo.

[krpeacock]

Ah right, that's the domain-specific stuff. I'll unignore that folder and push it up for reproduction to spare you from needing to install our SDK. You won't end up with a fully working example, but the frontend should compile

[krpeacock]

Should be good to go!

If you wanted to test the full stack example, the commands to set up would be

sh -ci "$(curl -fsSL https://sdk.dfinity.org/install.sh)"
dfx start --background
dfx deploy
npm start

[byCedric]

Awesome, thanks! When I add the following libraries:

$ npm install --save buffer text-encoding-polyfill

And add this to the top of the App.tsx file:

// Add TextEncoder polyfill
import 'text-encoding-polyfill';
// Add Buffer polyfill
import { Buffer } from 'buffer';
global.Buffer = Buffer;

It seems to work, without compiler issues. Only the warning that it can't connect to the network, which makes sense since I don't have that. 😄

[krpeacock]

That's super interesting! I'm still getting the export error on my machine when I try to run the web build. Were you compiling to iOS or Android?

[byCedric]

Ah, you are compiling to web! For me, it works on both Android and iOS. Haven't tested web, but I know what the problem could be. On web, we are using a plain webpack configuration to build the app. It does not transpile node_modules because we expect modules for the browser to be pre-transpiled for the browser. I don't think the majority of web browsers support the new module syntax, which means you have to manually add the module to the babel list to let it transpile.

To do this, you have to run $ expo customize:web and change the webpack config to this:

const createExpoWebpackConfigAsync = require('@expo/webpack-config');

module.exports = async function (env, argv) {
  const config = await createExpoWebpackConfigAsync(
    {
      ...env,
      babel: {
        dangerouslyAddModulePathsToTranspile: [
          // Ensure that all packages starting with @dfinity are transpiled.
          '@dfinity',
        ],
      },
    },
    argv
  );
  return config;
};

You might be better of including a pre-transpiled copy of the code, or have users add this configuration.

[krpeacock]

Great, this sounds viable then with some documentation updates!

As an ideal solution, Is there a way we could build a pre-transpiled version in addition to the harmony exports and signal that to Expo via the package.json? Or would you recommend we create an additional pre-transpiled package to publish to npm?

[byCedric]

Yes, that's also possible! I'm not sure how Snack would react, but we can always find a solution for that later.

• With normal npm modules, you should be able to define main file as entrypoint (defaults to index.js). This could contain plain Common JS code.
• With tools like Rollup and Webpack, you can define a module entrypoint. (although this isn't an "official" spec, it's still in proposal state). This could contain the ESM code.
• With React Native, you can define a react-native entry point. (couldn't find the docs for this, but you can see it there in action). This could contain the Common JS code, with some polyfills included.

All of these can be defined in the package.json, and there are a few like typings for TS. This guide might help making sense of some of these.

Other companies using Web3 opted for a separate React Native package. This is also fine and worth taking a look at, e.g. Magic Link