open-cli-tools
escaping pass-through arguments is too eager
Background
When trying to use concurrently with --passthrough-arguments option, any passed arguments get escaped too eagerly.
The issue has been observed while developing an application when trying to concurrently run a local web server and gherkin-testcafe for testing with the latter requring tags to start with @ character.
Since that application can't be shared here for legal reasons I've tried a simpler scenario which is reproducing the issue, too.
Scenario
- windows platform
- npm 10.5.2
- node 20.13.1
- concurrently 8.2.2
- files:
- package.json contains scripts:
{"scripts": { "foo": "concurrently -k -n \"bar,baz\" -P npm:foo:bar \"npm:foo:baz -- {@}\"", "foo:bar": "node block.js", "foo:baz": "node log.js", }} - file block.js looks like this:
setTimeout(() => console.log("delayed"), 1000); - file log.js looks like this:
console.dir(process.argv);
- package.json contains scripts:
Test
invoke without arguments
$ npm run foo
> [email protected] foo
> concurrently -k -n "bar,baz" -P npm:foo:bar "npm:foo:baz -- {@}"
[bar]
[bar] > [email protected] foo:bar
[bar] > node block.js
[bar]
[baz]
[baz] > [email protected] foo:baz
[baz] > node log.js
[baz]
[baz] [
[baz] 'C:\\Program Files\\nodejs\\node.exe',
[baz] 'C:\\Users\\cepharum\\PhpstormProjects\\UID\\Logistics\\WebClient\\Source\\UiNew\\log.js'
[baz] ]
[baz] npm run foo:baz -- exited with code 0
--> Sending SIGTERM to other processes..
[bar] npm run foo:bar exited with code 1
looks good ... setup is working as expected
invoke with arguments to pass
$ npm run foo -- -- --tags=@foo
> [email protected] foo
> concurrently -k -n "bar,baz" -P npm:foo:bar "npm:foo:baz -- {@}" -- --tags=@foo
[bar]
[bar] > [email protected] foo:bar
[bar] > node block.js
[bar]
[baz]
[baz] > [email protected] foo:baz
[baz] > node log.js --tags\=\@foo
[baz]
[baz] [
[baz] 'C:\\Program Files\\nodejs\\node.exe',
[baz] 'C:\\Users\\cepharum\\PhpstormProjects\\UID\\Logistics\\WebClient\\Source\\UiNew\\log.js',
[baz] '--tags\\=\\@foo'
[baz] ]
[baz] npm run foo:baz -- --tags\=\@foo exited with code 0
--> Sending SIGTERM to other processes..
[bar] npm run foo:bar exited with code 1
Problem here:
- the assignment operator gets escaped
- the
@gets escaped
Escaping seems to be eligible due to cmd.exe being involved. But eventually that escaping is found in invoked sub-process, too. Trying to fix it in that sub-process is not an option for it is a third-party application (gherkin-testcafe).
gherkin-testcafe illustrates the provision of tags without assignment operator, so I tried that one, too:
$ npm run foo -- -- --tags @foo
> [email protected] foo
> concurrently -k -n "bar,baz" -P npm:foo:bar "npm:foo:baz -- {@}" -- --tags @foo
[baz]
[baz] > [email protected] foo:baz
[baz] > node log.js --tags \@foo
[baz]
[bar]
[bar] > [email protected] foo:bar
[bar] > node block.js
[bar]
[baz] [
[baz] 'C:\\Program Files\\nodejs\\node.exe',
[baz] 'C:\\Users\\cepharum\\PhpstormProjects\\UID\\Logistics\\WebClient\\Source\\UiNew\\log.js',
[baz] '--tags',
[baz] '\\@foo'
[baz] ]
[baz] npm run foo:baz -- --tags \@foo exited with code 0
--> Sending SIGTERM to other processes..
[bar] npm run foo:bar exited with code 1
The @ character still ends up escaped.
I've also tried using {*} on invoking concurrently. So, the script in package.json is now:
"foo": "concurrently -k -n \"bar,baz\" -P npm:foo:bar \"npm:foo:baz -- {*}\"",
Invoking as before prevents the escaping, but yields a different issue:
$ npm run foo -- -- --tags @foo
> [email protected] foo
> concurrently -k -n "bar,baz" -P npm:foo:bar "npm:foo:baz -- {*}" -- --tags @foo
[baz]
[baz] > [email protected] foo:baz
[baz] > node log.js '--tags @foo'
[baz]
[bar]
[bar] > [email protected] foo:bar
[bar] > node block.js
[bar]
[baz] [
[baz] 'C:\\Program Files\\nodejs\\node.exe',
[baz] 'C:\\Users\\cepharum\\PhpstormProjects\\UID\\Logistics\\WebClient\\Source\\UiNew\\log.js',
[baz] "'--tags",
[baz] "@foo'"
[baz] ]
[baz] npm run foo:baz -- '--tags @foo' exited with code 0
--> Sending SIGTERM to other processes..
[bar] npm run foo:bar exited with code 1
So, the {*} seems to wrap all additional arguments in a single pair quotes. This may be intentional, but it definitely doesn't help here:
$ npm run foo -- -- --tags @foo --debug-on-fail
> [email protected] foo
> concurrently -k -n "bar,baz" -P npm:foo:bar "npm:foo:baz -- {*}" -- --tags @foo --debug-on-fail
[baz]
[baz] > [email protected] foo:baz
[baz] > node log.js '--tags @foo --debug-on-fail'
[baz]
[bar]
[bar] > [email protected] foo:bar
[bar] > node block.js
[bar]
[baz] [
[baz] 'C:\\Program Files\\nodejs\\node.exe',
[baz] 'C:\\Users\\cepharum\\PhpstormProjects\\UID\\Logistics\\WebClient\\Source\\UiNew\\log.js',
[baz] "'--tags",
[baz] '@foo',
[baz] "--debug-on-fail'"
[baz] ]
[baz] npm run foo:baz -- '--tags @foo --debug-on-fail' exited with code 0
--> Sending SIGTERM to other processes..
[bar] npm run foo:bar exited with code 1
I've just cloned concurrently locally to create a build off current main branch, then linked it into that other project.
$ git clone https://github.com/open-cli-tools/concurrently.git
$ cd concurrently
$ npm i
$ npm build
$ npm link
Then, in my project:
$ npm link concurrently
The outcome is the same. ~~I thought it might have been related to that windowVerbatimArguments option set. But nope.~~ I'm a fool ...
I've kept on trying to nail down the issue. From concurrently perspective, relying on shell-quote to do the job seems to cause this issue. As pointed out in the linked issue (comment1 comment2), maybe shell-quote isn't meant to support Windows' cmd.exe. What about concurrently? ... just asking.
Hey, thanks for the report.
Yes, concurrently should work on Windows. We have, however, been naive not to test with certain special characters such as @ and =, like you did.
I'm willing to be more lenient with escaping/wrapping in quotes than the shell-quote package if it means having less bugs like yours. E.g. arguments with a space need wrapping, unless they're already wrapped, of course. But I'm not sure about the others, might need to go through the list below and reimplement it in concurrently:
https://github.com/ljharb/shell-quote/blob/9eecafc0486c9321be223415cf3fb76a5bd07dda/quote.js#L14
(that said, I'm admittedly ignorant of risks of getting this wrong 😅 )
Well, I was hesitant to provide a solution I found based on my follow-up investigations as documented in the related issue at shell-quote repository for I was hoping to get a solution done there. However, since I wasn't able to wait for that with the original issue I was facing I've created a patched version of concurrently with shell-quote being replaced with my package. Maybe, I should provide it as a PR here.
I ran into this same issue.
This is the concurrently command I have in my package.json:
"build": "npx concurrently --passthrough-arguments \"npm:build-cms -- {@}\" \"npm:build-lms -- {@}\" -- ",
When used with the command npm run build -- --env=baseUrl='/' I get the following output:
❯ npm run build -- --env=baseUrl='/'
> [email protected] build
> npx concurrently --passthrough-arguments "npm:build-cms -- {@}" "npm:build-lms -- {@}" -- --env=baseUrl=/
[build-cms]
[build-cms] > [email protected] build-cms
[build-cms] > cd ./CMS && npx webpack --env\=baseUrl\=/
[build-cms]
[build-lms]
[build-lms] > [email protected] build-lms
[build-lms] > cd ./LMS && npx webpack --env\=baseUrl\=/
[build-lms]
[build-cms] [webpack-cli] Error: Unknown option '--env\=baseUrl\=/'
[build-lms] [webpack-cli] Error: Unknown option '--env\=baseUrl\=/'
[build-cms] [webpack-cli] Did you mean '--env'?
[build-lms] [webpack-cli] Did you mean '--env'?
[build-cms] [webpack-cli] Run 'webpack --help' to see available commands and options
[build-lms] [webpack-cli] Run 'webpack --help' to see available commands and options
[build-lms] npm run build-lms -- --env\=baseUrl\=/ exited with code 2
[build-cms] npm run build-cms -- --env\=baseUrl\=/ exited with code 2
Note how = is escaped and the ' characters are removed.
~~In my case, wrapping the pass-through argurments in double quotes results in everything being passed-through as expected:~~
Wrapping the pass-through arguments in double quotes results in everything looking like it would be ok:
❯ npm run build -- "--env=baseUrl='/'"
> [email protected] build
> npx concurrently --passthrough-arguments "npm:build-cms -- {@}" "npm:build-lms -- {@}" -- --env=baseUrl='/'
[build-lms]
[build-lms] > [email protected] build-lms
[build-lms] > cd ./LMS && npx webpack --env=baseUrl='/'
[build-lms]
[build-cms]
[build-cms] > [email protected] build-cms
[build-cms] > cd ./CMS && npx webpack --env=baseUrl='/'
[build-cms]
[build-cms] Base URL has been set to: '/'
[build-cms] API URL has been set to: '/'api
but the single quotes are erroneously being passed to the sub-command. When I run the webpack command directly in the relevant directory you can see the correct usage of the slash:
❯ npx webpack --env=baseUrl='/'
Base URL has been set to: /
API URL has been set to: /api
Version information:
Node: 18.20.8 npm: 10.8.2 concurrently: 8.2.2 PowerShell 7.4.7 Windows Terminal 1.22.10731.0
I haven't had much time since reporting this issue and there is a PR I have to take another look into though basically it is working as I have tested myself in a fork which has been in use at our site since then.
No problem. I just found this github issue while investigating my issue and wanted to document the issue on my side.
I actually updated my comment because my workaround didn't actually resolve the problem.