Add AWS::Include functionality to Fn::Sub

[benkehoe]

There are a lot of large documents that exist on CloudFormation resources. API Gateway swagger, definitions for Step Functions, CodePipeline, and CodeBuild, etc.

It would be convenient to store the document as a separate file. The AWS::Include transform exists to help with this, but it does not suffice for two reasons:

• The syntax of a file used with AWS::Include is still CloudFormation. These documents are the value of a single property, and have their own internal syntax; the file should just contain the document, not the surrounding CloudFormation property key (which is what's required for AWS::Include); this would let the files undergo validation that doesn't need to understand CloudFormation at all.
• The document must reference other resources in the rest of the template (for example, the Lambda function ARNs inside a state machine), but now you have a sort of circular reference problem: you have to keep this document in sync with changes to the template, for example if a logical id of a resource changes in the template, how does the developer making that change know if that resource is referenced in any included documents?

I have been asking individual teams to add the functionality in their APIs to retrieve these documents from S3, and additionally to add environment variable-like along with it, so that the document need not be hard-coded with certain references. But maybe CloudFormation could add a native capability to solve this.

Suppose Fn::Sub was extended to allow retrieving the template string from S3, performing the substitution as well as optionally parsing the result into the Json CloudFormation type (so the document could be either JSON or YAML independent of the template's syntax). The "environment variable" aspect comes from the existing semantics of Fn::Sub. The SAM CLI could understand an Fn::Sub with a CodeUri and automatically perform the upload.

What this would look like for the example in the AWS::CodePipeline::Pipeline resource docs (link):

The template snippet:

AppPipeline: 
  Type: AWS::CodePipeline::Pipeline 
  Properties: 
    RoleArn:
      !Ref: CodePipelineServiceRole 
    Stages: 
      !SubS3
        - s3://my-bucket/my-stages.yaml
        - SourceS3Bucket: !Ref MyBucket
          SourceS3ObjectKey: "my-key"
          ApplicationName: !Ref ApplicationName
          DeploymentGroupName: !Ref DeploymentGroupName
    ArtifactStore: 
      Type: S3 
      Location:
        Ref: ArtifactStoreS3Location 
      EncryptionKey:
        Id: arn:aws:kms:useast-1:ACCOUNT-ID:key/KEY-ID
        Type: KMS
    DisableInboundStageTransitions: 
      - 
        StageName: Release 
        Reason: "Disabling the transition until integration tests are completed"
    Tags:
      - Key: Project
        Value: ProjectA
      - Key: IsContainerBased
        Value: 'true'

The contents of s3://my-bucket/my-stages.yaml:

- Name: Source 
  Actions: 
  - Name: SourceAction
    ActionTypeId: 
      Category: Source 
      Owner: AWS 
      Version: 1 
      Provider: S3 
    OutputArtifacts: 
      - Name: SourceOutput 
    Configuration: 
      S3Bucket: ${SourceS3Bucket}
      S3ObjectKey: ${SourceS3ObjectKey}
    RunOrder: 1 
- Name: Beta 
  Actions: 
  - Name: BetaAction 
    InputArtifacts: 
      - Name: SourceOutput 
    ActionTypeId: 
      Category: Deploy 
      Owner: AWS 
      Version: 1 
      Provider: CodeDeploy
    Configuration: 
      ApplicationName: ${ApplicationName}
      DeploymentGroupName: ${DeploymentGroupName}
    RunOrder: 1 
- Name: Release 
  Actions: 
  - Name: ReleaseAction
    InputArtifacts: 
      - Name: SourceOutput 
    ActionTypeId: 
      Category: Deploy 
      Owner: AWS 
      Version: 1
      Provider: CodeDeploy 
    Configuration: 
      ApplicationName: ${ApplicationName}
      DeploymentGroupName: ${DeploymentGroupName}
    RunOrder: 1 

[luiseduardocolon]

Hi Ben. I get what you're trying to do. I'm ok with seeing how many others see value in this too. I was wondering: would writing a macro be a cleaner way to do this? You can pass stuff into it too. Just wondering if you've thought about that angle.

[benkehoe]

Hm, maybe? Maybe it could be part of the SAM macro... paging @jlhood

[rjlohan]

I like the suggestion as it also helps me solve an ask that a few teams are coming up with for the CloudFormation Registry - which is 'let me model an arbitrary Map<String, Object>'. i.e; an arbitrary JSON Document as a property. I don't want to do this in the base resource layer as we lose the ability to validate or provide meaning to the input document, but externalising the content as a separate document solves that resource modelling problem. Which leaves me with; adding custom validation hooks for these documents so that you get an early warning if you malformed it.

[jasonmac01]

Some teams within Amazon use Jinja with an internal tool as part of their build chain that transforms the Jinja into a full CFN template. Jinja also supports referencing across (local) files.

We've had enjoyment and success managing this process, giving us the ability to create control structures and reduce code. For example, with CloudWatch dashboards, we can enumerate a list of variables and generate several ::Widgets in a dashboard. Same for alarms and custom metrics, which can be verbose.

Could we not generalize some sort of Jinja-like support? It sounds like that is what Ben is requesting - more of a drop-in replacement of YAML/JSON.

[benkehoe]

I don't think providing client-side tooling is a good answer, because processes that suffice inside Amazon are not sufficient for many customers. One of the primary advantages of CloudFormation is that it is a managed service. A build step in a CI/CD pipeline is not a managed service. Many AWS users don't have good CI/CD practices, so for them it's now happening on a developer's laptop. It's great that Amazon has figured out ways to make CloudFormation less verbose, but what you should do is bake that into CloudFormation itself.

[jasonmac01]

No no no, sorry for the confusion. I'm not advocating for a client-side solution in this case. I'm merely suggesting that this sort of lightweight, template-based injection is powerful and something to consider in the context of this discussion.

[bjorg]

Alternatively, a version of AWS::Include that allows loading of plain text and feeding it to Fn::Sub. Something like:

      !Sub
        - !IncludeText s3://my-bucket/my-stages.yaml
        - SourceS3Bucket: !Ref MyBucket
          SourceS3ObjectKey: "my-key"
          ApplicationName: !Ref ApplicationName
          DeploymentGroupName: !Ref DeploymentGroupName

It's a bit more specific than !SubS3 s3://my-bucket/my-stages.yaml which might act on the referenced document and have unintended side-effects when the document contains unintentional references.

[lejiati]

@benkehoe Again, really appreciate your feedback! I'm transferring this issue over to a new GitHub repository dedicated to CloudFormation template language issues.