Allow interactive commands through @nrwl/workspace:run-commands

[JakeGinnivan]

Description

Sometimes you just want to be able to wrap a CLI tool for a project without using a plugin. For example:

        "serve": {
            "executor": "@nrwl/workspace:run-commands",
            "options": {
                "commands": [
                    {
                        "command": "shopify app serve"
                    }
                ],
                "cwd": "apps/shopify-app"
            }
        }

When you run this, sometimes it prompts you

~/_code/shopify-example/apps/shopify-app feature/nx-serve-functions* ❯ shopify app serve                                                                                                                                                                        13:22:24
? Do you want to convert mystore.myshopify.com to a development store?
Doing this will allow you to install your app, but the store will become transfer-disabled.
Learn more: https://shopify.dev/tutorials/transfer-a-development-store-to-a-merchant#transfer-disabled-stores
 (Choose with ↑ ↓ ⏎)
> 1. yes
  2. no

But all I see is

Motivation

There are so many useful CLI tools out there which do not default to --non-interactive (or even have a non-interactive version) and when they prompt you can't actually see what is printed to the console.

Suggested Implementation

I think there are multiple execution layers, i'm not sure how to do this. Given a few pointers I could take a crack.

Alternate Implementations

[jskrzypek]

@JakeGinnivan I think it should be straightforward and just come down to patching in a stdio-forwarding mode to the run-commands implementation here (run-commands.impl.ts#L174-L219).

If you look there you'll see it uses child_process's exec() & execSync(), which buffer the stdout & stderr and then resolve a promise with the buffered stdio (or just return it for execSync).

For the purpose you describe, this would probably be achievable by just adding a mode that uses child_process's spawn() & spawnSync() instead. Since you don't actually want to do anything programmatic with the output (just forward it back to the shell, you could probably just pass 'inherit' to spawn()'s options.stdio, which will tell the child process to inherit it's stdio from the parent, runner process. This means that the stdin, stdout, & stderr are shared between the processes. The only blocker I can think of would be if nx is doing something fancy with the stdio of the processes to make them cacheable or to for the build-analytics (I am pretty new to nx so I know nothing of how those systems work).

[JakeGinnivan]

For the moment I have just created my own executor in tools.

 "executor": "./tools/executors/workspace:run-command",
            "options": {
                "command": "dotnet watch msbuild /t:RunFunctions",
                "cwd": "apps/shopify-functions"
            }

tools/executors/workspace/package.json
{
    "executors": "./executor.json"
}

tools/executors/workspace/executor.json
{
    "executors": {
        "run-command": {
            "implementation": "./run-command/impl",
            "schema": "./run-command/schema.json",
            "description": "Runs a command"
        }
    }
}

tools/executors/workspace/run-command/schema.json
{
    "$schema": "http://json-schema.org/schema",
    "type": "object",
    "cli": "nx",
    "properties": {
        "command": {
            "type": "string",
            "description": "The command to run"
        },
        "cwd": {
            "type": "string",
            "description": "The working directory to run the command in"
        }
    },
    "required": [
        "command"
    ]
}

tools/executors/workspace/run-command/impl.ts
import execa from 'execa'

export default async function buildExecutor(options: {
    command: string
    cwd?: string
}) {
    console.info(`Executing workspace:run-command...`)

    await execa.command(options.command, {
        cwd: options.cwd,
        stdio: [process.stdin, process.stdout, 'pipe'],
    })

    return { success: true }
}

Happy to dig in, but a few of my other PRs have been closed with please discuss with the team the solution first. So now I don't invest my time into contributing without getting acknowledgement from the team about the solution

[dereekb]

I ran into this issue. In my case, I need to run the command inside a docker container so the environment has access to mongodb/etc.

        "watch": {
          "executor": "@nrwl/workspace:run-commands",
          "options": {
            "command": "./exec-with-server.sh 'npx nx run server:run-tests --watch'",
            "color": true
          }
        },
        "run-tests": {
          "executor": "@nrwl/jest:jest",
          "outputs": ["coverage/packages/server"],
          "options": {
            "jestConfig": "packages/server/jest.config.js",
            "passWithNoTests": true
          }
        },

When I run the watch target it doesn't pass through the interactivity. I thought jest was acting up at first.

For now I just have bash file that runs the watch command as presented above, but it would be nice to have interactivity when using @nrwl/workspace:run-commands, or a similar command that supported interactivity.

[jskrzypek]

@JakeGinnivan that makes a lot of sense, execa is a nice library. I'm pretty new to the nx community so that's a little disappointing that they treat PRs that way, but maintaining open source is a lot of work, just like contributing to open source, so I get it :shrug:

In this case I do think there's totally room for the wrapper you built to be turned into a plugin that provides that executor. It wouldn't need to be merged with the core nx project that way... Let me know if that's something you're interested in doing and I'll see if I can help out!

[JakeGinnivan]

Yeah, that is my end goal. I actually don't use many of the core NX plugins because I use TypeScript project references, vitest and vite a lot. Time is the main blocker for getting things up and running.

[dustinsgoodman]

👍 to this issue. Thank you @JakeGinnivan for reporting and providing some workarounds!

Adding another use case and a bug this is causing:

• Intricate outputs from other CLIs can't forward their commands appropriately as nx is blocking causing important detail blocks to not be properly formatted. Example - serverless offline supplies a block of mapped routes when invoking that is getting muddled and makes the feature unusable:

  Actual:

  Expected:

• I've created a Node repl with history for testing (similar to a rails console). Unfortunately, the history is inaccessible as nx is blocking interactivity when run via run-commands so pressing the up arrow to go back in history causes the following result:

[dustinsgoodman]

Wanting to confirm that the workaround (quoted below) from @JakeGinnivan is a perfect solve to my above issues. Additionally, wanted to add a couple of tweaks/notes that would help others in the future using this executor:

1. Use execa version 5.1.1
2. I had to modify my impl.ts as follows to work:

import { commandSync } from 'execa';

export default async function buildExecutor(options: {
  command: string;
  cwd?: string;
}) {
  console.info(`Executing workspace:run-command...`);

  commandSync(options.command, {
    cwd: options.cwd,
    stdio: [process.stdin, process.stdout, 'pipe'],
  });

  return { success: true };
}

For the moment I have just created my own executor in tools.

 "executor": "./tools/executors/workspace:run-command",
            "options": {
                "command": "dotnet watch msbuild /t:RunFunctions",
                "cwd": "apps/shopify-functions"
            }

tools/executors/workspace/package.json
{
    "executors": "./executor.json"
}

tools/executors/workspace/executor.json
{
    "executors": {
        "run-command": {
            "implementation": "./run-command/impl",
            "schema": "./run-command/schema.json",
            "description": "Runs a command"
        }
    }
}

tools/executors/workspace/run-command/schema.json
{
    "$schema": "http://json-schema.org/schema",
    "type": "object",
    "cli": "nx",
    "properties": {
        "command": {
            "type": "string",
            "description": "The command to run"
        },
        "cwd": {
            "type": "string",
            "description": "The working directory to run the command in"
        }
    },
    "required": [
        "command"
    ]
}

tools/executors/workspace/run-command/impl.ts
import execa from 'execa'

export default async function buildExecutor(options: {
    command: string
    cwd?: string
}) {
    console.info(`Executing workspace:run-command...`)

    await execa.command(options.command, {
        cwd: options.cwd,
        stdio: [process.stdin, process.stdout, 'pipe'],
    })

    return { success: true }
}

Happy to dig in, but a few of my other PRs have been closed with please discuss with the team the solution first. So now I don't invest my time into contributing without getting acknowledgement from the team about the solution

[AgentEnder]

@JakeGinnivan For single command runs, or serial command runs, does this not already work? I.e. changing to use "command" instead of "commands", or adding "parallel: false" to your target. When running in parallel forwarding stdin doesn't make much sense, but in serial its doable (and the default behavior)

[dustinsgoodman]

@AgentEnder I don't think that's necessarily true. The run-commands implementation explicitly inherits its io settings from its parent process as seen here:

https://github.com/nrwl/nx/blob/efedd2eff78700a72bcc30bdf7450656860a4ffb/packages/workspace/src/executors/run-commands/run-commands.impl.ts#L212-L219

I tested a fork of this implementation and switched the stdio settings to:

stdio: [process.stdin, process.stdout, process.stderr],

which has yielded a working result.

[dereekb]

@AgentEnder It does not. My example above uses a single command and it does not work.

[robbesettlemint]

Anyone know why when trying to run this executor I'm getting this common error:

import { commandSync } from 'execa';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:352:18)
    at wrapSafe (node:internal/modules/cjs/loader:1026:15)
    at Module._compile (node:internal/modules/cjs/loader:1061:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1149:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:999:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at /Users/administrator/Repos/orange-ebsi/node_modules/nx/src/shared/workspace.js:122:28
    at /Users/administrator/Repos/orange-ebsi/node_modules/nx/src/commands/run.js:146:36

I mean normally u can fix this by adding for example "type": "module" but that didn't work. Does anyone know why I'm having this issue and u guys are not. Would it help to add a tsconfig?

[dustinsgoodman]

Anyone know why when trying to run this executor I'm getting this common error:

import { commandSync } from 'execa';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:352:18)
    at wrapSafe (node:internal/modules/cjs/loader:1026:15)
    at Module._compile (node:internal/modules/cjs/loader:1061:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1149:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:999:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at /Users/administrator/Repos/orange-ebsi/node_modules/nx/src/shared/workspace.js:122:28
    at /Users/administrator/Repos/orange-ebsi/node_modules/nx/src/commands/run.js:146:36

I mean normally u can fix this by adding for example "type": "module" but that didn't work. Does anyone know why I'm having this issue and u guys are not. Would it help to add a tsconfig?

You probably need to ensure tslib is installed correctly. Also what command are you running that's producing the error?

[robbesettlemint]

I'm trying to run nx build eks-cluster && pulumi up -s stackname --cwd apps/eks-cluster, but getting the same when trying echo hello so it's definitely not the command. also I have tslib installed in my root package.json: "tslib": "^2.0.0". and ran yarn again to be sure. So I don't think it's that either

[robbesettlemint]

I made it work by building the ts file and running the js file. Also changed it a bit so u can run multiple commands:

import { commandSync } from 'execa';

export default async function buildExecutor(options: {
  commands: string[];
  cwd?: string;
}) {
  console.info(`Executing workspace:run-command...`);

  options.commands.forEach(command => {
    commandSync(command, {
      cwd: options.cwd,
      stdio: [process.stdin, process.stdout, 'pipe'],
    });
  });


  return { success: true };
}

schema.json

{
    "$schema": "http://json-schema.org/schema",
    "type": "object",
    "cli": "nx",
    "properties": {
        "commands": {
            "type": "array",
            "description": "The commands to run"
        },
        "cwd": {
            "type": "string",
            "description": "The working directory to run the command in"
        }
    },
    "required": [
        "commands"
    ]
}

[JakeGinnivan]

@robbesettlemint make sure you install execa@^5 rather than the latest.

https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c latest version is ESM only

[notaphplover]

Hi @JakeGinnivan, thank you so much for your workaround. Regarding this, what are your thoughts about nx lifecycles? It seems the actual run and run-many ones only clear stdout lines, in my case is causing some trouble:

packages/nx/src/tasks-runner/life-cycles/dynamic-run-one-terminal-output-life-cycle.ts

const clearDependentTargets = () => {
  for (let i = 0; i < dependentTargetsNumLines; i++) {
    readline.moveCursor(process.stdout, 0, -1);
    readline.clearLine(process.stdout, 0);
  }
};

It seems these life cycles are not designed for this interactive approach, in my case I can successfully pipe stdio, but my output is cleared providing a poor experience.

I'm not an expert of the project but it seems nx@13 introduced some sort of life cycle API (I could not find any docs), so maybe there's still hope to deal with this in any way.

Edit: nvm, It seems NX_STREAM_OUTPUT env set to true does the trick. I'll let this comment in case someone faces the same issue.

[william-kerr]

@Coly010, thanks for linking that fix! I created a simple run-interactive-command executor and added "outputCapture": "direct-nodejs" in schema.json. Then, I was able to use execSync from child_process (instead of execa), like so:

execSync(command, {
  cwd: options.cwd,
  stdio: ['inherit', 'inherit', 'inherit'],
})

[zrisher]

+1 to fixing this in main. Appreciate the provided workarounds, but Nx should provide a command runner that does not swallow interactive output.

[fahimalizain]

Adding outputCapture: direct-nodejs in schema.json of the executor as suggested by @william-kerr enabled cli-select to work perfectly 👍🏼 It was missing out on options otherwise.

[enchorb]

Would be great to have a nx:run-interactive-commands executor provided by Nx directly rather than users having to create their own

[empeje]

I have a workaround for this.

1. Setup your interactive script in the package.json
2. Run your script using nx:run-script e.g. here's my case

    "deploy": {
      "executor": "nx:run-script",
      "outputs": ["{workspaceRoot}/coverage/{projectRoot}"],
      "options": {
        "cwd": "packages/hologram-bos",
        "script": "bos:deploy"
      }
    },

and id my script:

  "scripts": {
    "bos:deploy": "bos components deploy"
  }

[miluoshi]

The interactive commands were working until we updated from Nx 15.4.5 to 16.0.0.

I think this change might have broken it by removing the child process being created with inherited stdio.

Adding this line to run-commands.impl.ts fixes the issue for me:

process.stdin.pipe(childProcess.stdin);

[mattfysh]

thanks @JakeGinnivan for the nice solution! I'm using this to run docker-compose up and needed to attach stdin so the output would print in colour. Almost necessary when running a cluster as the colours indicate which container emitted each log line.

The only thing I find now is that when I want to bring down the cluster via CTRL+C, the NX process group is killed while docker-compose is still coming down, leading to some dangling log lines after control has been returned to the terminal

Does anyone know how this could be fixed? I've tried using exec and spawn, and then capturing SIGINT but couldn't get it orchestrated correctly

[enchorb]

This also breaks metro for React Native. 'R' to reload from the terminal no longer works because Nx swallows the input

[wardds]

Same issue here, we have a little CLI tool (using inquirer for interactive input). It uses executor nx:run-commands for a straightforward ts-node ./src/main.ts command, but it's broken on nx v16.0.3

[AlexJWayne]

When I upgraded to NX 16 prisma now thinks it's being run from a non-interactive prompt. See: https://www.prisma.io/docs/concepts/components/prisma-migrate/prisma-migrate-limitations-issues#prisma-migrate-in-non-interactive-environments

I've applied this patch hack and input does work in other scripts, but prisma still complains.

[crusoexia]

This would become a deal breaker.

[MinskLeo]

For the moment I have just created my own executor in tools.

 "executor": "./tools/executors/workspace:run-command",
            "options": {
                "command": "dotnet watch msbuild /t:RunFunctions",
                "cwd": "apps/shopify-functions"
            }

tools/executors/workspace/package.json
{
    "executors": "./executor.json"
}

tools/executors/workspace/executor.json
{
    "executors": {
        "run-command": {
            "implementation": "./run-command/impl",
            "schema": "./run-command/schema.json",
            "description": "Runs a command"
        }
    }
}

tools/executors/workspace/run-command/schema.json
{
    "$schema": "http://json-schema.org/schema",
    "type": "object",
    "cli": "nx",
    "properties": {
        "command": {
            "type": "string",
            "description": "The command to run"
        },
        "cwd": {
            "type": "string",
            "description": "The working directory to run the command in"
        }
    },
    "required": [
        "command"
    ]
}

tools/executors/workspace/run-command/impl.ts
import execa from 'execa'

export default async function buildExecutor(options: {
    command: string
    cwd?: string
}) {
    console.info(`Executing workspace:run-command...`)

    await execa.command(options.command, {
        cwd: options.cwd,
        stdio: [process.stdin, process.stdout, 'pipe'],
    })

    return { success: true }
}

Happy to dig in, but a few of my other PRs have been closed with please discuss with the team the solution first. So now I don't invest my time into contributing without getting acknowledgement from the team about the solution

Thanks for solution. I made this gist based on exactly this response for easier copy-paste. https://gist.github.com/MinskLeo/749311c117996cf88e5594e61a4bb333

[PaulMest]

I couldn't get any of these custom executors solutions to work with nx 16.8. When I wrap a CLI with execa or node-pty outside of nx, it works perfectly.

I've also posted a related question on Stack Overflow. If anybody happens to solve this (or if NX fixes the likely culprit), feel free to answer on SO as well.

[codescalar]

Prisma is a pain to use with NX because of this CLI interactivity issue