aws
Unable to pass complex JSON file with multiple nestings for assigning permission to the SNS topic to send messages to the SQS queue
Following command issued:
aws sqs set-queue-attributes --queue-url https://sqs.us-west-1.amazonaws.com/123456789012/MyQueue --attributes file:sqspolicy.json
Note that the JSON file (sqspolicy.json) was first created within the SQS console using the following instructions - Step 2. Give permission to the SNS topic to send messages to the SQS queue. (http://docs.aws.amazon.com/sns/latest/gsg/SendMessageToSQS.html#SendMessageToSQS.sqs.permissions)
Contents of JSON file:
{
"Version": "2008-10-17",
"Id": "arn:aws:sqs:us-west-1:123456789012:MyQueue/SQSDefaultPolicy",
"Statement": [
{
"Sid": "Sid1363024842688",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "SQS:SendMessage",
"Resource": "arn:aws:sqs:us-west-1:123456789012:MyQueue",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:sns:us-west-1:123456789012:MyTopic"
}
}
}
]
}
Hi, any idea when this will be fixed?
I tried this with the latest version. Here's the results:
Error parsing parameter --attributes, should be: --attributes key_name=string,key_name2=string Where valid key names are: Policy VisibilityTimeout MaximumMessageSize MessageRetentionPeriod ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible CreatedTimestamp LastModifiedTimestamp QueueArn ApproximateNumberOfMessagesDelayed DelaySeconds ReceiveMessageWaitTimeSeconds
Here's the version info: C:>aws --version aws-cli/0.15.0 Python/2.7.3 Windows/7
I think what you really want to be able to do is this:
aws sqs set-queue-attributes --queue-url <queue_url> --attributes Policy=file://sqspolicy.json
Unfortunately, we don't support the file:// syntax for values of embedded fields, only for the top-level fields. One solution would be to customize this command and turn all of the possible attribute names into first-class options. Then, you could do something like this:
aws sqs set-queue-attributes --queue-url <queue_url> --policy file://sqspolicy.json
Does that seem like a reasonable solution?
Something definitely needs to be done here, if just clarifying with a complete example in the documentation. I have been URL encoding as requested and still can't get past this error:
A client error (InvalidAttributeValue) occurred when calling the SetQueueAttributes operation: Invalid value for the parameter Policy.
Requiring form-url-encoding is making it overly difficult for the user. Let us specify the policy and do whatever encoding is required in the code.
When I try to use the JSON format listed in the "help" output, it returns the error
Error parsing parameter '--attributes': should be: --attributes key_name=string,key_name2=string
which kind of implies that it isn't even looking for a JSON alternative.
Hi there,
I do have the issue as well. As @garnaat specified enclosed [] properties (second level) are not being able to be parsed. As the policy must contains it, we can not automate the create of policy via CLI directly from a valid json file. Any idea when this is going to be possible to parse that?
Thanks :)
I think the best idea I've heard so far is here: https://github.com/aws/aws-cli/issues/1803#issuecomment-189027369, but we're open to other ideas.
We can probably consolidate feature requests and track this work over in #1803.
Good idea. Side note: for those who are looking for a solution, I got mine here: https://alestic.com/2014/12/s3-bucket-notification-to-sqssns-on-object-creation/ (with the modification specified in the comments)
I met the policy invalid type error,too. I converted policy dict to a json string and it was accepted. Hope that could help you.
By the way, I was calling boto3's api 'set_queue_attributes' function to do that.
As this is so completely brain-dead I have to add my nasty solution to save the next poor sot that ends up having to deal with this revolting API that someone in AWS actually signed off on:
cat >sqs.json <<-EOT
{
"Policy" : "{ \"Statement\" : [ { \"Action\" : \"SQS:SendMessage\", \"Effect\" : \"Allow\", \"Sid\": \"AllowPESends\", \"Principal\" : { \"AWS\" : \"*\" }, \"Condition\" : { \"ArnEquals\" : { \"aws:SourceArn\" : \"${sns_topic_arn}\" } }, \"Resource\" : \"${sqs_arn}\" } ], \"Id\" : \"SQSPESendPolicy\", \"Version\" : \"2012-10-17\" }"
}
EOT
aws sqs set-queue-attributes --region ${region} --queue-url ${sqs_url} --attributes file://sqs.json
Yes folks you have to embed a json file as a string, in ...wait for it... another json file, and there's more, write all that to a file, and put that file on the command line.
And, yes, you have to spec the Principal as "AWS":"*" because the account that sends the SQS message is not yours.
AWS has some cool stuff, but Jeez, this was not finished. Even my ugliest debug crap is leaner than that.
Oh man I need some help with this. Your example doesn't work for me, bedge, though I'm sure it's because I'm doing something different somehow, and the most this thing tells me is "Invalid value for the parameter Policy".
AWS just needs to have some mechanism for automatically applying the appropriate policy to a queue when you subscribe it to a topic. It should almost certainly just happen as part of the subscription process. This process is ridiculous.
Hey @TalosThoren, I eventually shoved the whole mess into a couple of shell functions.
The connect_sqs is the one you want, but I included the create_sqs_with_dead_drop it calls in case there's something else in there that I'm doing that you're not.
#
# SQS can have a dead leter SQS for messages that fail many times.
# This has to be created prior to the primary SQS
# This method creates both SQSs
# reauired param:
# $1 base_name
# optional params:
# $2 maxReceiveCount - before going to dead-letter sqs, default:10
# $4 visibility_timeout
# if eval'd, sets:
# sqs_name
# sqs_url
# sqs_name_dead
# sqs_url_dead
#
create_sqs_with_dead_drop() {
sqs_name=$1
max_receive_count=${2:-10}
visibility_timeout=${3:-60}
# Temp dir deleted on exit
tmpdir=$(mktemp -d /tmp/d.ec2-common.XXXXXXXX) && cd "$dir" || exit 99
trap 'rm -rf "$tmpdir"' EXIT
# SQS's can be recreated nondestructively
while true ; do
sqs_url_dead=$(aws sqs create-queue --queue-name "${sqs_name}-dead" \
--region $region --output text)
[ $DEBUG ] && sqs_dead_url=$sqs_dead_url
sqs_arn_dead=$(aws sqs get-queue-attributes --queue-url ${sqs_url_dead} \
--attribute-names QueueArn --region $region --output text | awk "{print \$2}") \
&& break
sleep 5
done
cat << EOF > ${tmpdir}/attributes-dead.json
{
"DelaySeconds": "10",
"MaximumMessageSize": "262144",
"MessageRetentionPeriod": "1209600",
"ReceiveMessageWaitTimeSeconds": "0",
"VisibilityTimeout": "60"
}
EOF
aws sqs set-queue-attributes --queue-url ${sqs_url_dead} --attributes file://${tmpdir}/attributes-dead.json --region $region
while true ; do
sqs_url=$(aws sqs create-queue --queue-name "${sqs_name}" \
--region $region --output text)
[ $DEBUG ] && sqs_url=$sqs_url
sqs_arn=$(aws sqs get-queue-attributes --queue-url ${sqs_url} \
--attribute-names QueueArn --region $region --output text | awk "{print \$2}") \
&& break
sleep 5
done
cat << EOF > ${tmpdir}/attributes.json
{
"DelaySeconds": "10",
"MaximumMessageSize": "262144",
"MessageRetentionPeriod": "1209600",
"ReceiveMessageWaitTimeSeconds": "0",
"RedrivePolicy": "{\"deadLetterTargetArn\":\"${sqs_arn_dead}\",\"maxReceiveCount\":\"${max_receive_count}\"}",
"VisibilityTimeout": "${visibility_timeout}"
}
EOF
aws sqs set-queue-attributes --queue-url ${sqs_url} --attributes file://${tmpdir}/attributes.json --region $region
cat << EOT
export sqs_url="${sqs_url}"
export sqs_name="${sqs_name}"
export sqs_arn="${sqs_arn}"
export sqs_url_dead="${sqs_url_dead}"
export sqs_name_dead="${sqs_name}-dead"
export sqs_arn_dead="${sqs_arn_dead}"
EOT
}
function connect_sqs() {
sns_arn=$1
sqs_name=$2
max_rc=$3
vis_to=$4
tmpfile=$(mktemp /tmp/sns_sub.sh.$?.XXXXXXXX)
eval $(create_sqs_with_dead_drop "${sqs_name}" $max_rc $vis_to)
# Poorly documented mechanism for allowing the prod publication-events topic to write to out sqs.
# Note the json encoded string as an embedded json parameter. Someone at AWS signed off on this??
cat >${tmpfile} <<-EOT
{
"Policy" : "{ \"Statement\" : [ { \"Action\" : \"SQS:SendMessage\", \"Effect\" : \"Allow\", \"Sid\": \"AllowPESends\", \
\"Principal\" : { \"AWS\" : \"*\" }, \"Condition\" : { \"ArnEquals\" : { \"aws:SourceArn\" : \"${sns_arn}\" } }, \
\"Resource\" : \"${sqs_arn}\" } ], \"Id\" : \"SQSPESendPolicy\", \"Version\" : \"2012-10-17\" }"
}
EOT
aws sqs set-queue-attributes --region ${region} --queue-url ${sqs_url} --attributes file://${tmpfile}
# Subscribe to prod's publication events
if ! aws sns subscribe --topic-arn ${sns_arn} --region ${region} \
--protocol sqs \
--notification-endpoint ${sqs_arn} ; then
cat <<-EOT
=============================================
SNS $1 TOPIC NOT ATTACHED
=============================================
EOT
return 1
fi
if [ ! "${NO_CLEAN_UP}" ] && [ "${PURGE}" ] ; then
aws sqs purge-queue --region ${region} --queue-url ${sqs_url} || true
fi
}
Good Morning!
We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.
This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.
As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.
We’ve imported existing feature requests from GitHub - Search for this issue there!
And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.
GitHub will remain the channel for reporting bugs.
Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface
-The AWS SDKs & Tools Team
Based on community feedback, we have decided to return feature requests to GitHub issues.
create_sqs_with_dead_drop() { sqs_name=$1 max_receive_count=${2:-10} visibility_timeout=${3:-60} # Temp dir deleted on exit tmpdir=$(mktemp -d /tmp/d.ec2-common.XXXXXXXX) && cd "$dir" || exit 99 trap 'rm -rf "$tmpdir"' EXIT cat << EOF > ${tmpdir}/attributes.json { "DelaySeconds": "10", "MaximumMessageSize": "262144", "MessageRetentionPeriod": "1209600", "ReceiveMessageWaitTimeSeconds": "0", "RedrivePolicy": "{\"deadLetterTargetArn\":\"${sqs_arn_dead}\",\"maxReceiveCount\":\"${max_receive_count}\"}", "VisibilityTimeout": "${visibility_timeout}" } EOF aws sqs set-queue-attributes --queue-url ${sqs_url} --attributes file://${tmpdir}/attributes.json --region $region cat << EOT export sqs_url="${sqs_url}" export sqs_name="${sqs_name}" export sqs_arn="${sqs_arn}" export sqs_url_dead="${sqs_url_dead}" export sqs_name_dead="${sqs_name}-dead" export sqs_arn_dead="${sqs_arn_dead}" EOT }
Spent two days debugging CLI args for setting redrive policy. Upgraded to latest CLI, pored over issues and docs, etc. This was the only thing that worked. Unbelievable that the arg format is still such a problem 10 years later. :(
