Feature request - expose Haul as a library

[larixer]

Thank you guys for your amazing work! The need for Webpack-friendly React Native packager is really strong since it is very flexible, number one bundler at the moment and the fact that React Native has its own incompatible packager raises many problems with code sharing between mobile frontend and web frontend or even backend of the same application.

I would like to open discussion about exposing Haul as a library with API for:

• Making React Native webpack config with Haul entry points, resolvers, loaders, babel addons
• Using Haul as a webpack middleware, in similar fashion as webpack-dev-middleware will enable developers to deeply integrate Haul with their Webpack projects in a programmatic way. Much like the Webpack itself can be attached in a programmatic way to project via webpack-dev-middleware and instantiating Webpack compiler instances directly from Node.js code, without the need for launching command line Node.js process.

This is needed for projects that already use Webpack in a programmatic way via its Javascript API and want to include Haul as well the same way.

[thymikee]

This is something we want to tackle right after the #287 PR

[thymikee]

See: https://github.com/callstack/haul/issues/326

[hedgepigdaniel]

I'm not sure that https://github.com/callstack/haul/issues/326 really addresses this issue - at least not with the example API as written.

I was about to create a new issue but I realised this is probably the same issue. So here is my restatement of the issue:

Describe the feature

Instead of providing a fully optimised black box which hides the webpack/babel setup as the default, instead provide plugins/loaders/servers including only the necessary features (by default) to allow the user to use webpack with react native.

For example, these are features that are not easy to set up yourself with webpack and react-native, and are needed to use webpack with react-native:

• source maps that work on react-native devices
• react-native development server (bundle serving, symbolicating, etc)
• handling non standard module resolution used by the react-native package
• providing consistent polyfills for different versions of react-native

These features are useful in certain scenarios, but are not necessary to get webpack working with react-native:

• Having multiple processes building for multiple targets
• Having an integrated terminal experience with internal scrolling, etc
• Caching the results of loaders
• Farming loaders out to multiple processes

None of these features are specific to react-native, so IMO they would be best published as separate tools that can be used with any webpack build pipeline.

Sketching out usage of a potential alternative bare bones API

(I'm glossing over a lot of issues here, but hopefully the rough idea is clear):

// babel.config.js
module.exports = (api) => {
  const isAndroid = api.env((envName) => envName.indexOf('android') !== -1);
  const isIos = api.env((envName) => envName.indexOf('ios') !== -1);
  return {
    presets: [
      ['@haul-bundler/babel-preset-react-native-0.59', { platform: isAndroid ? 'android' : 'ios' }],
    ],
  };
};

// webpack.config.js
module.exports = (env) => {
  entry: [...HaulPlugin.polyfills, `src/index.${env.platform}.js`],
  loaders: [
    {
      test: /\.(([j|t]sx?)|mynewlanguageextension)$/
      loader: 'babel-loader',
      options: {
        envName: env.platform,
      },
    }, 
  ],
  plugins: [
    new HaulPlugin(),
  ]
};

// devServer.js
import { devServer } from '@haul-bundler/dev-server';
import makeWebpackConfig from './webpack.config.js';

const config = makeWebpackConfig();
const server = devServer(config); // Like webpack-dev-server

There's no reason why extra optimisations like caching, multiple processes, fancy CLI UIs, etc can't be used in this model (As separate webpack plugins, configuration flags, etc) - but such extra features are optional and users can make the decision of whether the benefit is worth the maintenance cost.

Motivation

As a developer that uses Haul instead of metro and co, I am used to wrangling with webpack/babel configurations (I like webpack - that's why I use haul and not metro). I am maintaining a project with various interesting issues related to webpack and babel, and despite haul's best efforts to abstract away the webpack/babel configuration and make my life easy, I still need to understand and patch them from time to time - and this is made more difficult by Haul's abstraction of these things.

I would much prefer if Haul was a little more work to set up initially (e.g. creating your own webpack/babel configurations, updating them according to release notes on new haul versions), if it was also easier to understand and customise (e.g. adding loaders to handle different source code types, using my own methods of caching loaders, doing a react-native-web and react-native build at the same time, making my own dev server CLI, etc etc).

Many of the problems I have had with haul come from these features which I don't need or want. These problems have taken up lots of my time to track down and fix:

• Any problem that I wanted to fix with a debugger was difficult to fix because there are separate processes that run the build in development mode, or more recently run babel and associated caching. Instead of being a simple matter of attaching a debugger to the build process, it becomes a challenge in itself to track down the process that triggers the bug at the right time.
• I cannot copy and paste text from errors in haul start because the auto scrolling terminal disables it (usually instead I forego watching and wait for a full fresh build with haul bundle)
• If there are many errors in the build, I can't easily see the first one
• I cannot use CLI args to inject constants into the build (e.g. yarn run start --api-url=http://staging.acme.com) because the process that actually builds is not the process that I started to do the building
• Making any change to the webpack configuration is difficult because I have to mutate an already complex configuration, and brittle because new haul releases often change the configuration and therefore break my code that mutates it.

Related Issues

• Problems using custom loaders due to integrated multiprocess/caching setup for babel: https://github.com/callstack/haul/issues/700
• Web as a platform might be a matter of not using haul at all, or using only some of its configuration helpers: https://github.com/callstack/haul/issues/132
• Desire to avoid cryptic errors (perhaps by following a README and adding one plugin/option at a time to understand why the configuration breaks): https://github.com/callstack/haul/issues/478
• Node API would not need to anything already covered by the existing webpack APIs: https://github.com/callstack/haul/issues/326

[zamotany]

@hedgepigdaniel Thanks for this detailed comment and while I do agree and would like to see Haul being just a set of utilities for Webpack (loaders, plugins, polyfills, etc) and a plugin for @react-native-community/cli that would use you configs files (eg webpack.config.js and others) to run Haul instead of Metro, there are some problems with that approach that we've already experienced.

Most noticeably lack of understanding or knowledge about Webpack. In the beginning Haul required you to have webpack.config.js and provided a CLI that would use utilise this config. This led to many issues, about not Haul, but Webpack - how to transpilenode_modules for example. At some point it was maintenance nightmare to solve other peoples problems with Webpack (not Haul) that we made a decision to hide Webpack config for regular users.

If we switch to providing utilities to configure Webpack yourself, that would be awesome for advanced users like yourself, but we would be opening ourselves to a rabbit hole of mistargeted questions about Webpack, and saying that This is Webpack issue, not Haul. Please raise this issue on Webpack issue tracker or Stackoverflow is not a good DX.

Next problem is limited resource and bandwidth so make that happen. Currently I'm the only person maintaining Haul with limited time to do so. I would be happy to help and provide guidance for any external contributor to implement those features or refactor some code in a way that would allow those Webpack utilities to be used separately from Haul CLI.

I don't think currently we will be able to straightaway go with the changes/API you've proposed, we could refactor pieces that are related to webpack, to allows for this low-level customisations. We cannot just drop the Haul CLI, because of companies that already invested in Haul and provided sponsoring for few features. The path forward for Haul is to either have community that contributes with PRs or companies that are willing to sponsor the work in Haul.

---

Now in the ideal world, Haul would consist of only Webpack utilities and a small bundler API that would be used by @react-native-community/cli, so the bundle command and packager server will be handled by @react-native-community/cli and only the bundling itself will be a handled by Haul, by as I said without help of the community or companies I'm not sure if we get there.

cc: @thymikee @grabbou

[thymikee]

As a maintainer of @react-native-community/cli I fully support @zamotany's idea to utilize Haul (and Webpack) as a CLI plugin.

We were already thinking about refactoring the way we use bundlers, as it's pretty tightly scoped to Metro at the moment. By creating a unified interface for bundler interactions, any bundler, like Metro or Haul/Webpack, could create a suitable adapter. Some reference: https://github.com/react-native-community/cli/issues/46.

[hedgepigdaniel]

Thanks for your response and generally your efforts on Haul - its really appreciated.

I don't think currently we will be able to straightaway go with the changes/API you've proposed, we could refactor pieces that are related to webpack, to allows for this low-level customisations. We cannot just drop the Haul CLI, because of companies that already invested in Haul and provided sponsoring for few features. The path forward for Haul is to either have community that contributes with PRs or companies that are willing to sponsor the work in Haul.

That sounds really good. The idea of having a black box and of having webpack plugins etc are not mutually exclusive. If the current black box is refactored further such that it is nothing more than a preset configuration of other more general tools, then both kinds of users can be supported. Better still, a user could start with the black box and at a certain point make the decision to switch to their own configurations. It then becomes more of a marketing/documentation issue to encourage users to use the API that is best for their needs.

Now in the ideal world, Haul would consist of only Webpack utilities and a small bundler API that would be used by @react-native-community/cli, so the bundle command and packager server will be handled by @react-native-community/cli and only the bundling itself will be a handled by Haul, by as I said without help of the community or companies I'm not sure if we get there.

Speaking for myself, I'm not sure I can do huge quantities of work on Haul, but I'm happy to help out with small incremental changes if they are relevant to my use case. In some respects I think exposing smaller tools will help in the long term to attract community support for the project. Many times I've been interested in solving a problem in Haul but given up quite early because I didn't find an obviously self contained/separate piece of code that was responsible for the thing I needed to patch. Perhaps a good starting point is factoring out more of the various loaders/plugins that are used internally into separate packages in the monorepo (especially true for functionality that is useful outside react-native - e.g. caching/threading). That way over time the existing API can be maintained while slowly exposing the various parts as standard webpack tools.

Aside: I wonder if this feature would be useful to move the haul webpack configuration into a webpack plugin?: https://github.com/webpack/webpack/pull/10401