[heft] introduce shell script plugin

[robertf224]

Adding a shell script plugin for Heft (relevant past discussion about this here https://github.com/microsoft/rushstack/issues/3519). Whipped this up internally at my company a few weeks ago / figured I would port here. Could use help on (1) how to best add tests for something like this (2) where to document.

You can see the plugin schema here, rough idea is you give it a command you want to run, and what build stage you'd like to run the command in. You can also give it either a separate command to run in watch mode (which i.e. might be the same as your main command with --watch appended, obvi depends on the tool) or a list of globs to watch if the tool you're proxying doesn't have a native watch mode.

Example heft.json wrapping protoc from within our internal repo:

{
    "$schema": "https://developer.microsoft.com/json-schemas/heft/heft.schema.json",
    "extends": "../node_modules/@chapter/heft-rig/profiles/node/config/heft.json",
    "heftPlugins": [
        {
            "plugin": "@chapter/heft-shell-script-plugin",
            "options": {
                "scripts": [
                    {
                        "name": "grpc-codegen",
                        "stage": "pre-compile",
                        "command": "mkdir -p ./src/__generated__/proto && protoc --plugin=./node_modules/.bin/protoc-gen-ts_proto --ts_proto_opt=env=node,outputServices=grpc-js,esModuleInterop=true --ts_proto_out=./src/__generated__/proto --proto_path=./src/proto $(find ./src/proto -iname '*.proto')",
                        "watchGlobs": ["./src/proto/*.proto"]
                    }
                ]
            }
        }
    ]
}

[dmichon-msft]

This looks pretty solid for the current version of Heft; issue is we're pretty close to merging the major refactor of Heft to generalize the scheduling process and that'll require a rather different way of hooking into the pipeline, so it doesn't feel quite worth merging this only to immediately redo it for compatibility. See migration documentation here: https://github.com/microsoft/rushstack/blob/release/heft-0.50.0-rc/apps/heft/UPGRADING.md

Watch mode documentation is currently a bit out of date, but the non-watch case should be quite straightforward to migrate. For watch mode, the new version of Heft no longer supports processes just performing rebuilds willy-nilly and there will instead need to be a way for the process being invoked to communicate to the Heft host that it wants to rebuild, then be notified when it is safe to do so. This is to ensure that the build pipeline steps happen in the expected sequence when changes occur.

[robertf224]

ah gotcha, I'll just wait until that merges and refactor then 👍

for my understanding, this means when wrapping most tools you'll need to pay the cost of startup on each rebuild even if the tool has a native watch mode? obvi not a big deal for rust-based/etc. tools which seems to be where most of the ecosystem is headed, but for node-based tools might be more of a factor

[robertf224]

would you rather leave this PR open or close and I'll open a new one later for the redo?

[dmichon-msft]

Feel free to just leave it open.

[Faithfinder]

Let me do a "bump" here, now that heft upgrade is done