[macro] Refs do not work for CommaDelimitedList types

[jutley]

I would like to be able to create reusable CloudFormation templates that are customized through parameters. Part of this means that I would have a parameter that is a list of AWS Account IDs.

This does not work using this macro because the macro keeps the value as-is -- a Ref to the parameter. CloudFormation then fails because the generated Assignment contains a list of Account IDs instead of a single ID.

To get around this limitation, I would love if this macro resolved parameter refs by replacing them with the parameter value, and expanding comma-delimited lists if necessary. Then, the rest of the macro's code would generate each Assignment as usual.

Alternatively, the logic of replacing parameter refs with the parameter values can be done in an independent Transform. I'm not aware of any existing solutions for this.

[benkehoe]

This is interesting. In general I'd say OUs as targets tend to be better than parameterized lists of accounts, but of course that's not always possible. I'm not sure I can accomplish this, though. The macro does not receive the resolved parameter value, and does not know the length of the list, so I don't think the information needed to unroll the list is present.

[benkehoe]

@benbridts do you have any ideas as to how this could be accomplished?

[benkehoe]

(besides a new Fn::Map function, of course)

[jutley]

I completely forgot that OUs are supported. That should work in my case. If you would like to keep it open to explore the idea further, feel free. Otherwise it can be closed.

[benkehoe]

Note that OUs are expanded by the macro, so if you change the membership of the OU, you need to do a stack update, which will cause the macro to get the new membership of the OU

[jutley]

Oh, I'm not realizing the OUs do not fix this problem. A parameterized OU results in this error:

[ERROR]	2021-04-19T20:43:31.753Z	e5b3b56f-376e-43ab-b30b-607b30a6fcd3	An error occurred: 'dict' object has no attribute 'startswith'
Traceback (most recent call last):
  File "/var/task/aws_sso_util/cfn_lib/macro.py", line 227, in handler
    output_template, max_stack_resources, resource_collection_dict = process_template(input_template,
  File "/var/task/aws_sso_util/cfn_lib/macro.py", line 111, in process_template
    resource_collection = resources.get_resources_from_config(
  File "/var/task/aws_sso_util/cfn_lib/resources.py", line 413, in get_resources_from_config
    for account in ou_fetcher(ou, recursive=False):
  File "/var/task/aws_sso_lib/lookup.py", line 518, in lookup_accounts_for_ou
    ou_type = "root" if ou.startswith("r-") else "OU"
AttributeError: 'dict' object has no attribute 'startswith'

It's a very similar scenario as this issue's original problem: Ref's are not being resolved to their actual value before the Assignment expansion takes place.

[benkehoe]

Oh, of course. I don't see much way around that either.

[benbridts]

I haven't looked at the code, so there might be a reason this won't work, but could you detect when you are passed a !Ref (because it's a dict instead of a str) and get the parameter value from templateParameterValues? (And if the value is a lists loop over them)

templateParameterValues

Any parameters specified in the Parameters section of the template. AWS CloudFormation evaluates these parameters before passing them to the function.

(docs link)

[jutley]

I took a stab at implementing this with #30.

[jutley]

@benkehoe Has there been any progress on this? I haven't been following this project since I was last poking at it last April. The changelogs mention something around better support for CFn functions, but I couldn't find what specifically this was in reference to.

[benkehoe]

Thanks for the poke; I have not made progress on this. I have some other changes in progress with the macro, I will look into this again.

[andrewlytle]

Found this issue after trying to !Ref an OU id as a param. Might be good to just catch the error and spit back a more useful error message at least?

[iainelder]

My use case is that I may have to maintain multiple organizations. In theory each organization would have matching Identity Center setups. Of course the OU IDs would be different, so to avoid duplicate templates I want to parametrize the values.

@benkehoe , would you accept a pull request to address this? #30 was closed without a clear resolution.

A good start would be to implement the expected behavior below.

Environment:

• The macro is already deployed
• The organization has an OU ou-zzzz-zzzzzzzz that contains AWS accounts 111111111111 and 222222222222

Given a template file ou-ref-template.yaml:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS-SSO-Util-2020-11-08

Parameters:
  TargetOU1:
    Type: String
    Default: ou-zzzz-zzzzzzzz

Resources:
  Group:
    Type: SSOUtil::SSO::AssignmentGroup
    Properties:
      Principal:
        - Type: USER
          Id: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
      PermissionSet:
        - ps-aaaaaaaaaaaaaaaa
      Target:
        - Type: AWS_OU
          Id: {"Ref": "TargetOU1"}

And given a Bash function to give me the transformed template:

function test-macro {
    template_file="$1"

    aws cloudformation create-change-set \
    --stack-name MacroTest \
    --template-body "file://$template_file" \
    --change-set-name ChangeSet1 \
    --change-set-type CREATE \
    --capabilities CAPABILITY_AUTO_EXPAND \
    > /dev/null

    aws cloudformation wait change-set-create-complete \
    --stack-name MacroTest \
    --change-set-name ChangeSet1

    aws cloudformation get-template \
    --stack-name MacroTest \
    --change-set-name ChangeSet1 \
    --template-stage Processed \
    --query TemplateBody

    aws cloudformation delete-stack \
    --stack-name MacroTest

    aws cloudformation wait stack-delete-complete \
    --stack-name MacroTest
}

Instead of giving me the transformed template, the function shows that that CloudFormation fails to create the change set.

$ test-macro ou-ref-template.yaml

Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state: For expression "Status" we matched expected path: "FAILED"

An error occurred (ValidationError) when calling the GetTemplate operation: ChangeSet template with id ChangeSet1 and stack id MacroTest does not exist

The macro log gives the error AttributeError: 'dict' object has no attribute 'startswith'.

[ERROR]	2023-07-17T07:38:02.391Z	aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa	An error occurred: 'dict' object has no attribute 'startswith'
Traceback (most recent call last):
  File "/var/task/aws_sso_util/cfn_lib/macro.py", line 228, in handler
    output_template, max_stack_resources, resource_collection_dict = process_template(input_template,
  File "/var/task/aws_sso_util/cfn_lib/macro.py", line 112, in process_template
    resource_collection = resources.get_resources_from_config(
  File "/var/task/aws_sso_util/cfn_lib/resources.py", line 413, in get_resources_from_config
    for account in ou_fetcher(ou, recursive=False):
  File "/var/task/aws_sso_lib/lookup.py", line 526, in lookup_accounts_for_ou
    ou_type = "root" if ou.startswith("r-") else "OU"
AttributeError: 'dict' object has no attribute 'startswith'
END RequestId: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa

I expected to get output like this:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Parameters": {
        "TargetOU1": {
            "Type": "String",
            "Default": "ou-zzzz-zzzzzzzz"
        }
    },
    "Resources": {
        "GroupAssignmentF7CC77_1": {
            "Type": "AWS::SSO::Assignment",
            "Metadata": {
                "SSO": {
                    "AssignmentGroupResourceName": "Group"
                }
            },
            "Properties": {
                "InstanceArn": "arn:aws:sso:::instance/ssoins-aaaaaaaaaaaaaaaa",
                "PrincipalType": "USER",
                "PrincipalId": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",
                "PermissionSetArn": "arn:aws:sso:::permissionSet/ssoins-aaaaaaaaaaaaaaaa/ps-aaaaaaaaaaaaaaaa",
                "TargetType": "AWS_ACCOUNT",
                "TargetId": "111111111111"
            }
        },
        "GroupAssignmentF7CC77_2": {
            "Type": "AWS::SSO::Assignment",
            "Metadata": {
                "SSO": {
                    "AssignmentGroupResourceName": "Group"
                }
            },
            "Properties": {
                "InstanceArn": "arn:aws:sso:::instance/ssoins-aaaaaaaaaaaaaaaa",
                "PrincipalType": "USER",
                "PrincipalId": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",
                "PermissionSetArn": "arn:aws:sso:::permissionSet/ssoins-aaaaaaaaaaaaaaaa/ps-aaaaaaaaaaaaaaaa",
                "TargetType": "AWS_ACCOUNT",
                "TargetId": "222222222222"
            }
        }
    }
}

[iainelder]

Another use case for this: I want to test my infrastructure configuration in a sandbox organization before I deploy it to my production organization.

There's no easy way to do that when I must embed a literal OU ID in the template.

I could accept duplication for this part of the configuration and deploy one version or the other in my CI pipeline depending on whether the target is sandbox or production.

I could use a Jinja2 template to inject the correct value depending on the target. Sceptre supports this. This might be okay if I can store the sets of values in Sceptre's config. But one of the main selling points of the macro is to avoid templating, right? :slightly_smiling_face: