[RRFC] Pluggable script runners

[justinfagnani]

Motivation ("The Why")

There are a number of tools, and a few RRFCs issues here (https://github.com/npm/rfcs/issues/190, https://github.com/npm/rfcs/issues/610 and https://github.com/npm/rfcs/issues/548) that relate to running scripts in series, parallel, etc.

The existing RRFCs are in some ways relatively simple, but the area of how and when to run scripts is quite deep and complex, and beyond what order to run scripts in, quickly gets into things like:

• Dependency graphs of scripts
• Freshness-checking of script inputs
• Caching script outputs
• Displaying script console output
• Handling long-running ("service") scripts: dependencies between them, noticing when ready, when to restart

It seems like npm has a few options on how to proceed with more sophisticated script running, ranging from:

1. Do nothing and leave it up to external tools
2. Incrementally add more features, like serial and parallel scripts, and grow in complexity
3. Design and implement a full-featured script graph system akin to Wireit

While I would love to see a great script runner built-in, I think all three of these options have serious downsides.

I'd like to propose an alternative that should be simple and allow for external project to extend npm script running capabilities, possibly creating well-worn paths for npm to adopt later, which is pluggbale script runners.

A script runner would be registered in a package.json file and get to intercede on running all scripts.

Ex:

{
  "runner": "wireit",
  "scripts": {
    "build": "tsc",
    "start": "web-dev-server"
  },
  "devDependencies": {
    "wireit": "^0.9.5"
  }
}

Upon npm commands that invoke scripts, like npm start, npm test, npm run, etc., npm would call the runner with the script to run and command line flags.

Alone, this doesn't enable much (though a runner could have flags run scripts of monorepo dependencies, etc), so we need a way to add more metadata about scripts for runners to consume. Letting scripts be objects instead of just strings would solve this.

{
  "runner": "wireit",
  "scripts": {
    "build": {
      "command": "tsc",
      "inputs": ["src/**/*.ts"],
      "outputs": ["build/"]
    },
    "start": {
      "command": "web-dev-server",
      "dependencies": ["build"]
    }
  },
  "devDependencies": {
    "wireit": "^0.9.5",
  }
}

With this additional information the script runner can now check inputs for changes, cache build outputs, and run dependencies or check that they're fresh.

Certain script metadata fields can be standardized, most importantly "command" which would be supported by the built-in runner. Other fields would be metadata for specific runners.

Over time the "runner" field could be a place to opt-in to new built-in running behavior.

Note that this is essentially building in more ergonomic support for what's possible today if you delegate all script to a runner manually. Wireit works on package.json configs like this:

{
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "command": "tsc"
    }
  }
}

I think explicitly carving out space for runners would discourage some of the fragmentation in the ecosystem where many runners have their own CLI that side-steps npm and creates a new thing to learn and barrier to entry for new contributors. The script runner approach keeps things fairly well-hidden behind plain npm commands.

Example

See above.

How

Current Behaviour

Currently scripts only execute the command given, and don't run in series, parallel, or with dependencies. Custom script runners have to be manually setup with their own script definition sections in package.json.

Desired Behaviour

Allow runners to bring the features of tools like Turborepo, Nx, Wireit, etc, but keep the external interface of the npm CLI.

References

• https://github.com/npm/rfcs/issues/190
• https://github.com/npm/rfcs/issues/548
• https://github.com/npm/rfcs/issues/610

Alternatives

An alternative would be to build the features of, say Wireit, directly into npm. I actually think this would be a great outcome, especially for workspaces, but I've been assuming that it's a bit to big of an ask.