[Bug] Yarn shell throws errors about Glob patterns

[MrEfrem]

• [ ] I'd be willing to implement a fix

Describe the bug

Next.js uses so format of files: [id].jsx, [...subpaths].jsx. When I execute this command in the scripts section of package.json:

{
  "scripts": {
    "copyfile": "shx cp [id].jsx somefolder/[id].jsx",
  }
}

Yarn throws this error:

Internal Error: No file matches found: ".../[id].jsx". Note: Glob patterns currently only support files that exist on the filesystem (Help Wanted)
    at interpolateArguments (/Users/aleksandr/WebProjects/testproject/.yarn/releases/yarn-sources.cjs:47916:46)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async executeCommandChain (/Users/aleksandr/WebProjects/testproject/.yarn/releases/yarn-sources.cjs:48064:24)
    at async executeCommandLine (/Users/aleksandr/WebProjects/testproject/.yarn/releases/yarn-sources.cjs:48148:26)
    at async executeShellLine (/Users/aleksandr/WebProjects/testproject/.yarn/releases/yarn-sources.cjs:48189:25)
    at async execute (/Users/aleksandr/WebProjects/testproject/.yarn/releases/yarn-sources.cjs:48364:10)
    at async realExecutor (/Users/aleksandr/WebProjects/testproject/.yarn/releases/yarn-sources.cjs:47310:14)
    at async /Users/aleksandr/WebProjects/testproject/.yarn/releases/yarn-sources.cjs:47330:12
    at async NodeFS.mktempPromise (/Users/aleksandr/WebProjects/testproject/.yarn/releases/yarn-sources.cjs:28705:18)
    at async Module.executePackageScript (/Users/aleksandr/WebProjects/testproject/.yarn/releases/yarn-sources.cjs:47295:10)

I only find solution to replace the filename with ?id?.jsx.

2. I was had to extract this command from the scripts section into a bash script

{
  "scripts": {
    "gcp-build": "find build -name '\\[@@@*' -exec bash -c 'mv \"$1\" \"${1/@@@/...}\"' -- \\{\\} \\;"
  }
}

because it thrown this error:

Internal Error: No file matches found: "\{\}". Note: Glob patterns currently only support files that exist on the filesystem (Help Wanted)

In the bash script it looks as:

find build -name '\[@@@*' -exec bash -c 'mv "$1" "${1/@@@/...}"' -- {} \;

[paul-soporan]

Can you check if #1790 fixes any of your issues (yarn set version from sources --branch 1790)? It reworks glob patterns a bit because they were quite buggy before.

[MrEfrem]

@paul-soporan the first problem is fixed, great job! In the second problem I don't see anymore any errors but now my commands don't do anything. You can test it so:

1. Add scripts in package.json:

"step0": "find src -name '\\[...*' -exec bash -c 'mv \"$1\" \"${1/.../@@@}\"' -- \\{\\} \\;",
"step1": "find src -name '\\[@@@*' -exec bash -c 'mv \"$1\" \"${1/@@@/...}\"' -- \\{\\} \\;",

2. Create a file with name: [...ids].jsx.
3. Run yarn step0. It should rename [...ids].jsx to [@@@ids].jsx.
4. Run yarn step1. It should rename [@@@ids].jsx to [...ids].jsx. When I run these commands directly in my shell it works well. I.e. something is still broken in Yarn.

[royra]

Adding another use case which is broken on our repo (works on yarn v1).

package.json scripts:

{
  "clean": "rm -rf ./dist .*.cache"
}

If there are no files matching .*.cache, it throws:

No file matches found: ".*.cache". Note: Glob patterns currently only support files that exist on the filesystem (Help Wanted)
    at O (/Users/roy/salto_private/.yarn/releases/yarn-2.3.3.cjs:2:519009)

Also happens with master (which now includes #1790).

May I ask what is the reasoning behind the new behavior - parsing the command and expanding globs, instead of leaving it to the shell?

[arcanis]

Your case is expected, @royra. Use || true to silence the error if matching nothing is the intended behaviour.

May I ask what is the reasoning behind the new behavior - parsing the command and expanding globs, instead of leaving it to the shell?

Yarn 2 implements a basic shell in order to make most commands portable across all environments, Linux and Windows alike. As for a lot of things we tend to prefer being strict and failing early, hence why we upgrade into errors glob patterns that have no match (instead of passing them untransformed to the underlying tool, which could have dire consequences).

[royra]

@arcanis thanks for the explanation! I for one find it surprising that Yarn's shell behaves differently than the standard (which is not to fail if globs match zero files).

If anyone's interested in the "standard" behavior, I managed to get it by explicitly running a "regular" shell like so:

{
  "clean": "bash -c 'rm -rf ./dist .*.cache'",
}

[arcanis]

Shells tend to have different behaviours. For instance zsh does throw on zero-match glob patterns. At least now we guarantee that the behaviour will be consistent by default, and you can opt-out if you must 🙂

[matzeeable]

and you can opt-out if you must

@arcanis Thanks for your explanation. How can I opt out this behavior? I do not see any conditional in the interpolate functionality:

https://github.com/yarnpkg/berry/blob/95dd7d6fe3e2ddf460a79c529231151d27012044/packages/yarnpkg-shell/sources/index.ts#L412-L523

[arcanis]

How can I opt out this behavior?

Directly call the shell you want to use:

{
  "foo": "bash -c 'ls *'"
}

[matzeeable]

@arcanis Unfortunately, this does not work for us. We are developing our application within a Docker container, so we can ensure everyone in our team is using the same bash / environment. E.g. we have this script:

{
    "scripts": {
        "wp-wait": "timeout=500 docker exec -e REST_ENDPOINT=$(cd plugins && ls | head -1) ${WP_WAIT_CONTAINER:=$(yarn --silent docker-compose:name-wordpress)} bash -c 'while [ \"$(curl --insecure -s -o /dev/null -w ''%{http_code}'' localhost/wp-json/$REST_ENDPOINT/v1/plugin)\" != \"200\" ]; do sleep 2; done'"
    }
}

Running the above command brings up the following error:

Syntax Error: Expected ":-", ":-}", "}", or [$@*?#a-zA-Z0-9_\-] but ":" found (line 1, column 91).
    at Ce (/[...]/.yarn/releases/yarn-berry.cjs:55:343841)

We have about ~50 one-liner scripts in our package.json#scripts. We love the simplicity of scripts as you can turn individual tasks into a simple alias. I don't want to open a barrel and move this scripts into another file. A simple example of our OSS: https://github.com/devowlio/wp-react-starter/blob/2e19fea30c921c9509bd54abcf0601486c91f28e/package.json#L21-L53 . In the end, such scripts should not end up e.g. in a grunt task

At least now we guarantee that the behaviour will be consistent by default

This is totally a cool idea, but for our use case not suitable (and I think there are other users as well). Simple things are no longer possible cause yarn tries to parse the script instead of pipe it through the current shell.

I want to contribute

I never had a look at the coding of yarn, but I tried to figure out where we can skip parsing the script and pipe it to the current shell. Unfortunately, I did not find the part, can you please point me to the right direction?

A possible solution

Extend the .yarnrc.yml with the following parameter:

interpretScripts: true

By default, this is activated as yarn intents to this. But setting this to false would simply pipe the script to the current shell.

What do you think? 😊

[milesj]

This also causes DEBUG to be useless with NPM scripts, for example DEBUG=beemo:* <cmd>.

No matches found: "beemo:*"

Furthermore, is it possible to escape it? I have --workspaces=* which worked in v1, but does not in v2. Quoting it like --workspaces='*' or --workspaces=\\* works on nix machines, but not windows. Not sure how to proceed...

[merceyz]

Quoting it like --workspaces='' or --workspaces=\ works on nix machines, but not windows

Yarn ships with a portable shell so if it works on one OS it works on all of them https://github.com/yarnpkg/berry/tree/2b308be563f7da8ba928ebc04d4813ead1b162b6/packages/yarnpkg-shell

[milesj]

I think something else is happening in relation to Yarn 2. My scripts hang on windows until they timeout, but according to the debugs, they ran fine. But that's a discussion for another topic.

[CopyJosh]

This also causes DEBUG to be useless with NPM scripts, for example DEBUG=beemo:* <cmd>.

No matches found: "beemo:*"

Furthermore, is it possible to escape it? I have --workspaces=* which worked in v1, but does not in v2. Quoting it like --workspaces='*' or --workspaces=\\* works on nix machines, but not windows. Not sure how to proceed...

Running into this issue when trying to use DEBUG in the same manner...

[merceyz]

You need to quote it to avoid glob expansion

{
  "scripts": {
    "foo": "DEBUG='beemo:*' node -p process.env.DEBUG"
  }
}

[link89]

You need to quote it to avoid glob expansion

{
  "scripts": {
    "foo": "DEBUG='beemo:*' node -p process.env.DEBUG"
  }
}

I thought it was problem of zsh until I read this reply. Thank you!

[zmk-punchbowl]

Use || true to silence the error if matching nothing is the intended behaviour.

In case this helps anyone, appending || true to the command did solve our problem, by preventing the resulting "no matches found" error from stopping the execution of subsequent commands, however it does not "silence" the error. "No matches found" still shows up in the console. This is fine for our purposes, but it was causing me a bit of confusion as to whether the || true was working as intended.

[snebjorn]

This is rather unfortunate. I encountered this using [email protected] and [email protected] by executing rimraf *.tsbuildinfo which would fail with

No matches found: "*.tsbuildinfo"

Even with the || true workaround it looks like a bunch of errors are printed during build :(

"scripts": {
  "build": "yarn clean && ..."
  "clean": "rimraf *.tsbuildinfo || true"
}

I upgraded from [email protected] where rimraf *.tsbuildinfo was working without incident.

Is there a better way to handle this?

[merceyz]

Is there a better way to handle this?

See https://github.com/yarnpkg/berry/issues/1731#issuecomment-893709156

[snebjorn]

I had understood that comment as I needed the DEBUG= part. But it was just

"clean": "rimraf '*.tsbuildinfo'"

Much obliged!