PowerShell: Wrap example does not work

[chrischu]

Describe the bug I tried running the sample from https://mikefarah.gitbook.io/yq/v/v4.x/operators/create-collect-into-object#wrap-prefix-existing-object in PowerShell but I can't seem to get it to work.

Version of yq: 4.6.1 Operating system: windows Installed via: binary release

Input Yaml Concise yaml document(s) (as simple as possible to show the bug, please keep it to 10 lines or less) sample.yml:

name: Mike

Command The command you ran:

yq eval '{""wrap"": .}' sample.yml

Actual behavior

Error: Parsing expression: Lexer error: could not match text starting at 1:1 failing at 1:2.
        unmatched text: "`"

Expected behavior

wrap:
  name: Mike

Additional context I tried both the original code snippet with single ", but then found out that I apparently have to double the quotes in PowerShell, but that didn't not help either :(.

[mikefarah]

Have you seen this: https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell ?

[chrischu]

I tried both the original code snippet with single ", but then found out that I apparently have to double the quotes in PowerShell, but that didn't not help either :(.

As I said: yes.

[mikefarah]

Oh right sorry - no idea then, I got that advice from here https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_quoting_rules?view=powershell-7.1 for what it's worth - perhaps there's doc for whatever version of powershell you are using..

[phreed]

The work around is to quote in the following way.

.\yq.exe e '{\"wrap\": .}' sample.yaml

You can see that this is strictly a Powershell issue by the following example. There is a tool to help with this echoargs

 echoargs e '{\"wrap\": .}' sample.yaml

Arg 0 is <e>
Arg 1 is <{"wrap": .}>
Arg 2 is <.\examples\name.yaml>

Command line:
"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" e "{\"wrap\": .}" .\examples\name.yaml

echoargs is available as a chocolatey package choco install echoargs.

[phreed]

Here is a yq.ps1 that seems to do the necessary escaping.

Param(
    [string]$command,
    [string]$yamlFile
  )
$escCommand =  $command -replace '"','\"' -replace ' ',''
$evalArgs = @("eval", $escCommand, $yamlFile)

Write-Output ".\yq.exe"  $evalArgs
Start-Process ".\yq.exe"  -ArgumentList $evalArgs -Wait -NoNewWindow -PassThru

I am not sure about the removing whitespace bit. But, without it Start-Process splits the evalArgs into 4 arguments instead of 3. I would like to see evaluate_sequence_command.go have a log message in the evaluateSequence() function showing the individual arguments.

[phreed]

Here is a yq.ps1 that seems to act better.

Param(
    [string]$command,
    [string]$yamlFile
  )
$escCommand =  $command -replace '"','\"' 
$evalArgs = @("eval", "$escCommand", $yamlFile)

Write-Output ".\yq.exe"  $evalArgs
& ".\yq.exe"  $evalArgs

[ferdnyc]

@phreed

The work around is to quote in the following way.

.\yq.exe e '{\"wrap\": .}' sample.yaml

You can see that this is strictly a Powershell issue by the following example. There is a tool to help with this echoargs

 echoargs e '{\"wrap\": .}' sample.yaml

Arg 0 is <e>
Arg 1 is <{"wrap": .}>
Arg 2 is <.\examples\name.yaml>

Command line:
"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" e "{\"wrap\": .}" .\examples\name.yaml

echoargs is available as a chocolatey package choco install echoargs.

Thanks for the pointer on echoargs, that's great to have.

Here's the one thing that confuses me, though: At least in PowerShell 6 (7 doesn't work under remote sessions), the "Arg..." output of echoargs is identical for these two cases, yet only one of them works. And in fact, the message yq spits out in the failure case has changed in subtle but interesting ways from @chrischu's OP:

PS C:\Users\***> yq -M '{""wrap"": .}' sample.yml
Error: parsing expression: Lexer error: could not match text starting at 1:2 failing at 1:7.
        unmatched text: "\"wrap:"

PS C:\Users\***> echoargs e '{""wrap"": .}' sample.yml
Arg 0 is <e>
Arg 1 is <{"wrap": .}>
Arg 2 is <sample.yml>

Command line:
"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" e "{""wrap"": .}" sample.yml

PS C:\Users\***> yq -M e '{\"wrap\": .}' sample.yml
wrap:
  name: Mike
PS C:\Users\***> echoargs e '{\"wrap\": .}' sample.yml
Arg 0 is <e>
Arg 1 is <{"wrap": .}>
Arg 2 is <sample.yml>

Command line:
"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" e "{\"wrap\": .}" sample.yml

In fact, another working form — as I just discovered — is this:

PS C:\Users\***> yq -M e "{\""wrap\"": .}" sample.yml
wrap:
  name: Mike

It doesn't work without the backslashes, and interestingly, echoargs also requires the backslashes to show the argument correctly, which is making me a bit suspicious of it now:

PS C:\Users\***> echoargs e "{""wrap"": .}" sample.yml
Arg 0 is <e>
Arg 1 is <{wrap: .}>
Arg 2 is <sample.yml>

Command line:
"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" e "{"wrap": .}" sample.yml

PS C:\Users\***> echoargs e "{\""wrap\"": .}" sample.yml
Arg 0 is <e>
Arg 1 is <{"wrap": .}>
Arg 2 is <sample.yml>

Command line:
"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" e "{\"wrap\": .}" sample.yml

(Notice how, without the backslashes, echoargs drops one set of quotes from its "Command line:" string, then the other set of quotes for its "Arg 1" output.)

[bschapendonk]

Powershell 7.4 breaks the behavior of "", a "seems to work just fine now

[chrischu]

If you want the old behavior you can use $PSNativeCommandArgumentPassing = "Legacy". See Passing arguments that contain quote characters

[chrischu]

But yq does some really weird stuff when using --%:

.\yq_windows_amd64.exe --% --verbose -n '.test = "something"'

prints

08:16:49 processArgs [DEBU] processed args: ['.test = "something"']
08:16:49 maybeFile [DEBU] checking ''.test' is a file
08:16:49 maybeFile [DEBU] error: CreateFile '.test: The system cannot find the file specified.
08:16:49 maybeFile [DEBU] result: false
08:16:49 processArgs [DEBU] assuming expression is ''.test'
Error: cannot pass files in when using null-input flag

So yq gets the correct parameters but for some reason thinks .test is a file.

[Aankhen]

I believe yq is getting the literal string '.test as its first argument, not just .test, because the stop-parsing token (--%) prevents PowerShell from parsing the quoted string into a single argument. See the difference in the processed args here:

❯❯ echo '' | yq --% --verbose -n .
13:03:51 processArgs [DEBU] processed args: [.]
13:03:51 maybeFile [DEBU] checking '.' is a file
13:03:51 maybeFile [DEBU] error: <nil>, dir: true
13:03:51 maybeFile [DEBU] result: false
13:03:51 processArgs [DEBU] assuming expression is '.'
13:03:51 FormatFromFilename [DEBU] using default inputFormat 'yaml'
13:03:51 initCommand [DEBU] Using input format yaml
13:03:51 initCommand [DEBU] Using output format yaml
13:03:51 ParseExpression [DEBU] Parsing expression: [.]
13:03:51 handleToken [DEBU] processing SELF (55)
13:03:51 handleToken [DEBU]   adding token to the fixed list
13:03:51 ConvertToPostfix [DEBU] postfix processing currentToken SELF (55)
13:03:51 ConvertToPostfix [DEBU] put SELF (55) onto the opstack
13:03:51 ConvertToPostfix [DEBU] postfix processing currentToken )
13:03:51 popOpToResult [DEBU] popped SELF (55) from opstack to results
13:03:51 ConvertToPostfix [DEBU] opstackLen: 0
13:03:51 ConvertToPostfix [DEBU] PostFix Result:
13:03:51 ConvertToPostfix [DEBU] > SELF
13:03:51 createExpressionTree [DEBU] pathTree SELF
13:03:51 GetMatchingNodes [DEBU] Processing Op: SELF
13:03:51 GetMatchingNodes [DEBU] D0, P, ScalarNode (!!null)::0 kids
13:03:51 PrintResults [DEBU] PrintResults for 1 matches
13:03:51 PrintResults [DEBU] -- print sep logic: p.firstTimePrinting: false, previousDocIndex: 0
13:03:51 PrintResults [DEBU] D0, P, ScalarNode (!!null)::0 kids
13:03:51 Encode [DEBU] encoderYaml - going to print D0, P, ScalarNode (!!null)::0 kids

13:03:51 PrintResults [DEBU] done printing results

❯❯ echo '' | yq --% --verbose -n '.'
13:03:54 processArgs [DEBU] processed args: ['.']
13:03:54 maybeFile [DEBU] checking ''.'' is a file
13:03:54 maybeFile [DEBU] error: CreateFile '.': The system cannot find the file specified.
13:03:54 maybeFile [DEBU] result: false
13:03:54 processArgs [DEBU] assuming expression is ''.''
13:03:54 FormatFromFilename [DEBU] using default inputFormat 'yaml'
13:03:54 initCommand [DEBU] Using input format yaml
13:03:54 initCommand [DEBU] Using output format yaml
13:03:54 ParseExpression [DEBU] Parsing expression: ['.']
Error: 1:1: invalid input text "'.'"