runs from command line not in shell script

[rojomisin]

Sorry if this is an easy one, banging my head on the wall at the moment... I can run this command on the command line, but once this is executed from sh/bash in OS X it fails

This works fine from command line jq --sort-keys '.TemplateBody | .Parameters.AMIpacker.Default = "ami-1234addd" '

shell script is

aws cloudformation get-template --stack-name $stack_name --profile $aws_profile | $JQPATH --sort-keys '.TemplateBody' >"$stack_name-current_$MYDATESTAMP.json"

### doesn't work in double quotes either
#JQSUBCMD="jq -r --sort-keys \".TemplateBody | .Parameters.$cfparam_name.Default = \\\"$cfparam_value\\\"\""

### single quote escaping 
JQSUBCMD='jq -r --sort-keys '"'"' .TemplateBody | .Parameters.'"$cfparam_name"'.Default = "'"$cfparam_value"'" '"'"' '

echo $JQSUBCMD
aws cloudformation get-template --stack-name $stack_name --profile $aws_profile >"$stack_name-target_$MYDATESTAMP.json"
$JQSUBCMD "$stack_name-target_$MYDATESTAMP.json"

Resulting output which echos the command and trys to run it

sh cloudformation_update_stack.sh stack pro AMIpacker ami-1234addd 
jq --sort-keys '.TemplateBody | .Parameters.AMIpacker.Default = "ami-1234addd" '
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
'.TemplateBody
jq: 1 compile error

Why doesn't this work in a shell script?

My use-case is to substitute or merge values into cloudformation templates using shell scripts (and jenkins jobs).

[glennj]

You can't store the command in a variable like this:

JQSUBCMD='jq -r --sort-keys '"'"' .TemplateBody | .Parameters.'"$cfparam_name"'.Default = "'"$cfparam_value"'" '"'"' '
....
$JQSUBCMD filename.

The jq body that you so carefully quoted will not remain as a single argument due to shell word splitting. You are actually trying to execute this command:

jq -r --sort-keys "'.TemplateBody" "|" ".Parameters.AMIpacker.Default" "=" "\"ami-1234addd\"" "'"

See http://mywiki.wooledge.org/BashFAQ/050 for the gory details.

If you absolutely need to store the command in a variable, use a bash array:

jqCmd=( jq -r --sort-keys ".TemplateBody | .Parameters.$cfparam_name.Default = \"$cfparam_value\"" )
....
"${jqCmd[@]}" "$filename"

[rojomisin]

Fair enough, and I would not normally do that, however I cannot get a shell variable to parse properly in jq. I guess my problem is that I cannot get the simplest use case of shell variable substitution to work in a script.

I think the problem lies in the fact that both shell and jq use the $ for variables.

why doesn't this work?

jq --sort-keys \'".TemplateBody | .Parameters.$cfparam_name.Default = \"ami-1234addd\""\' file.json

jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
'.TemplateBody | .Parameters.MyCentOSAMIpacker.Default = "ami-1234addd"'
jq: 1 compile error

Would you have an example for the simplest way to do this?

[wtlangford]

Assuming $cfparam_name is a shell variable, something like this should work for you:

jq --arg cfparam_name "$cfparam_name" --sort-keys '.TemplateBody | .Parameters.[$cfparam_name].Default = "ami-1234addd"' file.json

The trick, here, is to rebind your shell variable to a jq variable. We always recommend the use of single quotes on your jq program, to prevent your shell from attempting to expand jq variables. I put quotes around "$cfparam_name" when rebinding it to prevent spaces from breaking things.

On Wed, Apr 6, 2016 at 4:21 PM rojomisin [email protected] wrote:

Fair enough, and I would not normally do that, however I cannot get a shell variable to parse properly in jq. I guess my problem is that I cannot get the simplest use case of shell variable substitution to work in a script.

I think the problem lies in the fact that both shell and jq use the $ for variables.

why doesn't this work?

jq --sort-keys '".TemplateBody | .Parameters.$cfparam_name.Default = "ami-1234addd""' file.json

jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at , line 1: '.TemplateBody | .Parameters.MyCentOSAMIpacker.Default = "ami-1234addd"' jq: 1 compile error

Would you have an example for the simplest way to do this?

— You are receiving this because you are subscribed to this thread.

Reply to this email directly or view it on GitHub https://github.com/stedolan/jq/issues/1124#issuecomment-206542868

[rojomisin]

hi @wtlangford I cant see the example, I only see ... elipsis

[wtlangford]

Well, that's what I get for using my phone. Let's try again. Assuming $cfparam_name is a shell variable, something like this should work for you:

jq --arg cfparam_name "$cfparam_name" --sort-keys '.TemplateBody | .Parameters.[$cfparam_name].Default = "ami-1234addd"' file.json

The trick, here, is to rebind your shell variable to a jq variable. We always recommend the use of single quotes on your jq program, to prevent your shell from attempting to expand jq variables. I put quotes around "$cfparam_name" when rebinding it to prevent spaces from breaking things.

[rojomisin]

Yes, and this I have also tried, which is why it's so puzzling that it does not work (OS X el capitan jq-1.5)

I am switching the jq arg to be cfp to keep it straight, but I did try your example verbatim and it also failed the same way

jq --arg cfparam_name "$cfparam_name" --sort-keys '.TemplateBody | .Parameters.[$cfparam_name].Default = "ami-1234addd"' FusionNSqa-target_040612232016.211459970601.json
jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
.TemplateBody | .Parameters.[$cfparam_name].Default = "ami-1234addd"                            
jq: 1 compile error

shell substitution happens first, and then I would expect jq to be able to parse the arg variable cfp but this does not work, does it work for you?

I also tried double quotes to get it to interpolate but it also does not seem to work

enclosing the whole thing in "" just echoes a well-formed string, not execute jq on the string

jq --arg cfparam_name "$cfparam_name" --sort-keys "\".TemplateBody | .Parameters.$cfparam_name.Default = "ami-1234addd"\"" FusionNSqa-target_040612232016.211459970601.json
".TemplateBody | .Parameters.MyCentOSAMIpacker.Default = ami-1234addd"

[wtlangford]

Can I get the output of jq --version, please. I see you said 1.5, I just want to confirm.

On Wed, Apr 6, 2016 at 5:34 PM rojomisin [email protected] wrote:

Yes, and this I have also tried, which is why it's so puzzling that it does not work (OS X el capitan jq-1.5)

I am switching the jq arg to be cfp to keep it straight, but I did try your example verbatim and it also failed the same way

jq --arg cfparam_name "$cfparam_name" --sort-keys '.TemplateBody | .Parameters.[$cfparam_name].Default = "ami-1234addd"' FusionNSqa-target_040612232016.211459970601.json jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at , line 1: .TemplateBody | .Parameters.[$cfparam_name].Default = "ami-1234addd" jq: 1 compile error

shell substitution happens first, and then I would expect jq to be able to parse the arg variable cfp but this does not work, does it work for you?

I also tried double quotes to get it to interpolate but it also does not seem to work

enclosing the whole thing in "" just echoes a well-formed string, not execute jq on the string

jq --arg cfparam_name "$cfparam_name" --sort-keys "".TemplateBody | .Parameters.$cfparam_name.Default = "ami-1234addd""" FusionNSqa-target_040612232016.211459970601.json ".TemplateBody | .Parameters.MyCentOSAMIpacker.Default = ami-1234addd"

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/stedolan/jq/issues/1124#issuecomment-206580832

[rojomisin]

 jq --version
jq-1.5

[jontrainor]

I think I'm having a similar issue where this line in my script: echo $ASSESSMENT_RUN_STATUS | jq --arg arn "$ASSESSMENT_RUN_ARN" '.failedItems.$arn.failureCode'

results in the following error: jq: error: syntax error, unexpected '$', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1: .failedItems.$arn.failureCode

EDIT: I think I may have found a workaround for this issue. changing my command like this did not error:

echo $ASSESSMENT_RUN_STATUS | jq --arg arn "$ASSESSMENT_RUN_ARN" '.failedItems | .[$arn] | .failureCode'

[ruchikaguptaa]

I have similar issue in makefile target build.

BUILD_DATE := $(shell date -u +"%Y%m%d")
BUILD_TIME := $(shell date -u +"%H%M%S")
CHART_VERSION_OVERRIDE := "1.${BUILD_DATE}.${BUILD_TIME}"
$(BUILD_DIR)/abc:
	chart_meta=$(shell cat ../../ccp/charts/kube-control-plane/Chart.yaml | jq --arg version "$(CHART_VERSION_OVERRIDE)" '.version = $(version)')
	echo $(chart_meta) > ../../ccp/charts/kube-control-plane/Chart.yaml

But I get the following error. Any recommendations on how I can fix this?

jq: error: syntax error, unexpected $end (Unix shell quoting issues?) at <top-level>, line 1:
.version =           
jq: 1 compile error

[Uriel-Carrasquilla]

Thank you for this solution. I also found it hard to solve. I had to use a variable passed to my script Here is how I solved it: #!/bin/bash CC=DE

long string so I broke it up. I needed to plug "DE" in the string.

STR1='.results[] | select (.country==' STR2=' and .is_disabled==false) | ' STR3='{id:.id,cc:.country} ' CMD=( /usr/bin/jq -c "$STR1"$CC"$STR2$STR3") #echo ${CMD[@]} #exit

execute the command

"${CMD[@]}" page*.json