Replace RequireJS with ES6 modules

[laurenwalker]

[robyngit]

MetacatUI currently uses RequireJS for dependency management. As part of our effort to benefit from newer JavaScript features and improve performance, we want to migrate to ES6 modules. The task involves refactoring all the existing Backbone models, views, and collections to remove the usage of RequireJS and adopt ES6 module syntax.

Examples

Importing Dependencies

Before (RequireJS)

Here's how we currently import dependencies using RequireJS:

define(["backbone", "underscore", "jquery"], function(Backbone, _, $) {
  // Your code here
});

After (ES6 Modules)

With ES6, we should use the import keyword:

import Backbone from 'backbone';
import _ from 'underscore';
import $ from 'jquery';

Exporting Modules

Before (RequireJS)

With RequireJS, exporting often looks something like this:

define(["backbone"], function(Backbone) {
  var MyModel = Backbone.Model.extend({
    // model code here
  });
  return MyModel;
});

After (ES6 Modules)

In ES6, we use the export keyword. We can choose to export functions, objects, or classes as named or default exports:

// Define your model
var MyModel = Backbone.Model.extend({
  // model code here
});

// Export your model
export default MyModel;

Above we mix ES6 export syntax with traditional Backbone model definitions using Backbone.Model.extend(). We can go one step further and switch to ES6 classes:

class MyModel extends Backbone.Model {
  // model code here
}

export default MyModel;

Or, for named exports:

export class MyModel extends Backbone.Model {
  // model code here
}

Dynamic Imports

Before (RequireJS)

RequireJS allows us to dynamically load dependencies using require():

if (someCondition) {
  require(['./myModule.js'], function(myModule) {
    myModule.someFunction();
  });
}

After (ES6 Modules)

ES6 allows us to dynamically load dependencies using import():

if (someCondition) {
  import('./myModule.js')
    .then((myModule) => {
      myModule.someFunction();
    })
    .catch((error) => {
      console.log(`An error occurred: ${error}`);
    });
}

As shown above, an advantage of this syntax is that we can define what should happen if the import fails.

Alternatively, we can use await to wait for the import to resolve:

if (someCondition) {
  const myModule = await import('./myModule.js');
  myModule.someFunction();
}

Importing HTML Templates (text files)

Before (RequireJS)

In views, we import our HTML templates using the RequireJS text plugin:

define(["text!templates/myTemplate.html"], function(myTemplate) {
  // Template text is available in the myTemplate variable
});

After (ES6 Modules)

We can use the fetch() API to import HTML templates:

const myTemplate = await fetch('./templates/myTemplate.html').then((response) => response.text());

Alternatively, we can move toward including our templates in our JavaScript files using template literals:

const myTemplate = `
  <div class="my-template">
    <h1>${myVariable}</h1>
  </div>
`;

This is a more modern approach that allows us to avoid making additional HTTP requests for our templates, and is inline with component-based JavaScript frameworks like React.

Circular Dependencies

In at least three places in MetacatUI, we run into circular dependencies:

• Query Rule view
• Citations collection
• Filters collection

For example, the Filters collection requires the Filter Groups model, and Filter Groups model require the Filters collection. (This is because the Filter Groups can be nested, and each Filter Group contains a collection of Filters.) We work around this by using RequireJS's require() function where the dependency is needed, rather than at the top of the file:

var FilterGroup = require("models/filters/FilterGroup");
var newFilterGroup = new FilterGroup(attrs, options);
return newFilterGroup;

In ES6, when a circular dependency is detected, the JavaScript engine continues loading where it left off rather than restarting. The import and export statements are hoisted, ensuring that all modules are eventually executed and their exports become available to each other.

Therefore, we can simply import the dependency at the top of the file:

import FilterGroup from 'models/filters/FilterGroup';

Alternatively, we can use import() to dynamically load the dependency:

const FilterGroup = await import('models/filters/FilterGroup');

However, circular dependencies can lead to unexpected behavior, so we should test thoroughly and consider refactoring to avoid them where possible.

Tests

We currently use the Mocha framework for running unit and integration tests, along with the Chai assertion library.

Before (RequireJS)

Models, views, and collections are imported for testing using RequireJS:

define([
  "../../../../../../../../myModel",
], function (MyModel) {

  var should = chai.should();
  var expect = chai.expect;

  describe("MyModel Test Suite", function () {
    // Tests here
  });
});

After (ES6 Modules)

We will need to update our tests to use the import syntax:

import MyModel from '../../../../../../../../myModel';

const should = chai.should();
const expect = chai.expect;

describe("MyModel Test Suite", function () {
  // Tests here
});

Index.html

The <script> tags in the main index.html page will need to be updated to use type="module" to enable ES6 module syntax. We might also want to reconsider how MetacatUI.onStartGetLoader works. This function dynamically adds a new script tag to load loader.js, but we could use import instead. Also, for the MetacatUI object creation and its associated methods (MetacatUI.onStartConfigError, MetacatUI.onStartLoaderError, etc.), we can consider moving them to a dedicated ES6 module and then import that module in the main application file.

"use strict"

ES6 modules are in strict mode by default: When we convert to using import and export, we are automatically in strict mode and no longer need to declare "use strict" at the top of files. We can therefore delete all "use strict" statements from views, models, and collections once they've been converted.

Other Considerations

• Build Tools: Should we have integrated a bundler by the time we start this task, we'll need to update our build tools to support ES6 modules.
• Testing: We don't yet have full test coverage for MetacatUI, so we'll need to budget time to write tests before the migration or at least to thoroughly test the application after the migration.

Resources

• ES6 Modules in Depth
• How to Use Promises
• Javascript Modules & Cyclic Dependencies
• ECMAScript modules in browsers
• Backbone with ES6

[mbjones]

Another fantastic overview, thanks @robyngit . Quick question: would switching to ES6 mean contracting our browser version support, or would all of the browser versions that we currently support continue to work fine under ES6?

[robyngit]

Thanks @mbjones! ES6 modules are well-supported in modern web browsers, with all major browser engines having support since 2018. See Can I use, the ECMAScript compatibility table and MDN JavaScript modules. If we were trying to support IE 11 or older browsers, we could use a transpiler like Babel to compile ES6 into ES5.

[robyngit]

I did a quick experiment with automating converting MetacatUI JS files from using RequireJS to import/export syntax. It's only a rough experiment at the moment, but the code can be found here: https://github.com/robyngit/convert-metacatui-es6. With the exception of a couple of files that follow an unusual pattern, I think it will be possible to do a majority of the conversion automatically. I expect testing and debugging will take the most time.

[ianguerin]

With quite a few errors to sort through, I've tried using this NPM package to automate the translation as well https://www.npmjs.com/package/amd-to-es6

Could at least be useful to compare the results with those from Robyn's python script?

[robyngit]

Great to see the progress on transitioning MetacatUI to ES6 modules @ianguerin. Smart to use an existing library for the conversion if it works!!

One issue that I didn't consider initially, is that we're managing our third-party dependencies manually. Some are edited manually (!!) and some are quite old. These older dependencies might still rely on the CommonJS require() syntax or other non-module patterns. It might be worth doing an audit of our third-party dependencies to identify which ones could pose compatibility issues.

For those that are not ES6-compatible, we may have a few options:

• Look for modern alternatives that offer similar functionality and are ES6-module friendly.
• Explore if there are updated versions or forks of these dependencies that support ES6.
• Consider maintaining certain parts of the application using the require() syntax, specifically for these dependencies.

See also: https://github.com/NCEAS/metacatui/issues/2193

[ianguerin]

Using a require JS babel plugin, I was able to create a quick example of how ES6 modules and RequireJS modules can coexist, this could allows us to slowly migrate portions of the code rather than waiting for a big-bang migration. Here's the commit with a simple example of a few imports and exports!

[robyngit]

This is great @ianguerin! Love an iterative approach. Where do we use babel here? Is the suggestion to also include a build step on release (such that there's a /dist/ of metacatui)?

[ianguerin]

There's not actually a babel compilation step here, just uses 2 babel plugins to resolve paths and load files. I'm sure there's a way to do this without babel, but this was the easiest ready-made solution that I could find. If I understand the premise of your question though, I think there should be a build step so that we don't have to ship this new code to the user's browser as well, should happen in a compile time step and then excluded from the built JS file.

[robyngit]

Note/reminder: Once we have implemented linting & formatting (#2096), we'll need to consider how to update ESLint to accommodate both ES6 modules and commonJS. If we explicitly name files that use ES6 modules with the extension .mjs, this would make the configuration more simple.

For example, a potential snippet for eslint.config.mjs:

  // Configuration for CommonJS files (those using require)
  { 
    files: ["src/**/*.js"],
    languageOptions: { 
      sourceType: "commonjs"
    }
  },
  // Configuration for ES Modules files (those using import/export)
  { 
    files: ["src/**/*.mjs"],
    languageOptions: { 
      sourceType: "module"
    }
  },