Ansible Role: Create cloudformation stacks

Motivation | Installation | Features | Variables | Usage | Templates | Diff | Dependencies | Requirements | License

Ansible role to render an arbitrary number of Jinja2 templates into Cloudformation files and deploy any number of stacks.

Motivation

This role overcomes the shortcomings of Cloudformation templates itself as well as making heavy use of Ansible's features.

1. Cloudformation limitations - The Cloudformation syntax is very limited when it comes to programming logic such as conditions, loops and complex variables such as arrays or dictionaries. By wrapping your Cloudformation template into Ansible, you will be able to use Jinja2 directives within the Cloudformation template itself, thus having all of the beauty of Ansible and still deploy via Cloudformation stacks.
2. Environment agnostic - By being able to render Cloudformation templates with custom loop variables you can finally create fully environment agnostic templates and re-use them for production, testing, staging and other environments.
3. Dry run - Another advantage of using Ansible to deploy your Cloudformation templates is that Ansible supports a dry-run mode (--check) for Cloudformation deployments (since Ansible 2.4). During that mode it will create Change-sets and let you know what would change if you actually roll it out. This way you can safely test your stacks before actually applying them.

This role can be used to either only generate your templates via cloudformation_generate_only or also additionally deploy your rendered templates. So when you have your deployment infrastructure already in place, you can still make use of this role, by only rendering the templates and afterwards hand them over to your existing infrastructure.

When templates are rendered, a temporary build/ directory is created inside the role directory. This can either persist or be re-created every time this role is run. Specify the behaviour with cloudformation_clean_build_env.

Installation

Either use Ansible Galaxy to install the role:

$ ansible-galaxy install cytopia.cloudformation

Or git clone it into your roles directory

$ git clone https://github.com/cytopia/ansible-role-cloudformation /path/to/ansible/roles

Features

• Deploy arbitrary number of Cloudformation templates
• Create Cloudformation templates with Jinja2 templating engine
• Render templates only and use your current infrastructure to deploy
• Dry-run via Ansible --check mode which will create temporary Change sets (e.g.: lets you know if a resource requires re-creation)
• Have line-by-line diff between local and deployed templates via cloudformation_diff module
• Make use of Ansible vault to store sensitive information encrypted

Variables

Overview

The following variables are available in defaults/main.yml and can be used to setup your infrastructure.

Details

This section contains a more detailed describtion about available dict or array keys.

cloudformation_defaults

cloudformation_stacks

Examples

Define default values to be applied to all stacks (if not overwritten on a per stack definition)

# Enforce that 'profile' must be set for each cloudformation stack item
cloudformation_required:
  - profile

cloudformation_defaults:
  region: eu-central-1

Define cloudformation stacks to be rendered and deployed

cloudformation_stacks:
  - stack_name: stack-s3
    template: files/cloudformation/s3.yml.j2
    profile: production
    template_parameters:
      bucketName: my-bucket
    tags:
      env: production
  - stack_name: stack-lambda
    template: files/cloudformation/lambda.yml.j2
    profile: production
    termination_protection: True
    template_parameters:
      lambdaFunctionName: lambda
      handler: lambda.run_handler
      runtime: python2.7
      s3Bucket: my-bucket
      s3Key: lambda.py.zip
    tags:
      env: production

Only render your Jinja2 templates, but do not deploy them to AWS. Rendered cloudformation files will be inside the build/ directory of this role.

$ ansible-playbook play.yml -e cloudformation_generate_only=True

Usage

Simple

Basisc usage example:

playbook.yml

- hosts: localhost
  connection: local
  roles:
    - cloudformation

group_vars/all.yml

# Define Cloudformation stacks
cloudformation_stacks:
  # First stack
  - stack_name: stack-s3
    profile: testing
    region: eu-central-1
    template: files/cloudformation/s3.yml.j2
    template_parameters:
      bucketName: my-bucket
    tags:
      env: testing
  # Second stack
  - stack_name: stack-lambda
    profile: testing
    termination_protection: True
    region: eu-central-1
    template: files/cloudformation/lambda.yml.j2
    template_parameters:
      lambdaFunctionName: lambda
      handler: lambda.run_handler
      runtime: python2.7
      s3Bucket: my-bucket
      s3Key: lambda.py.zip
    tags:
      env: testing

Advanced

Advanced usage example calling the role independently in different virtual hosts.

inventory

[my-group]
infrastructure  ansible_connection=local
application     ansible_connection=local

playbook.yml

# Infrastructure part
- hosts: infrastructure
  roles:
    - cloudformation
  tags:
    - infrastructure

# Application part
- hosts: application
  roles:
    - some-role
  tags:
    - some-role
    - application

- hosts: application
  roles:
    - cloudformation
  tags:
    - application

group_vars/my-group.yml

stack_prefix: testing
boto_profile: testing
s3_bucket: awesome-lambda

cloudformation_defaults:
  profile: "{{ boto_profile }}"
  region: eu-central-1

host_vars/infrastructure.yml

cloudformation_stacks:
  - stack_name: "{{ stack_prefix }}-s3"
    template: files/cloudformation/s3.yml.j2
    template_parameters:
      bucketName: "{{ s3_bucket }}"
    tags:
      env: "{{ stack_prefix }}"

host_vars/application.yml

cloudformation_stacks:
  - stack_name: "{{ stack_prefix }}-lambda"
    template: files/cloudformation/lambda.yml.j2
    template_parameters:
      lambdaFunctionName: lambda
      handler: lambda.run_handler
      runtime: python2.7
      s3Bucket: "{{ s3_bucket }}"
      s3Key: lambda.py.zip
    tags:
      env: "{{ stack_prefix }}"

Templates

This section gives a brief overview about what can be done with Cloudformation templates using Jinja2 directives.

Example: Subnet definitions

The following template can be rolled out to different staging environment and is able to include a different number of subnets.

Ansible variables

---
# file: staging.yml
vpc_subnets:
  - directive: subnetA
    az: a
    cidr: 10.0.10.0/24
    tags:
      - name: Name
        value: staging-subnet-a
      - name: env
        value: staging
  - directive: subnetB
    az: b
    cidr: 10.0.20.0/24
    tags:
      - name: Name
        value: staging-subnet-b
      - name: env
        value: staging

---
# file: production.yml
vpc_subnets:
  - directive: subnetA
    az: a
    cidr: 10.0.10.0/24
    tags:
      - name: Name
        value: prod-subnet-a
      - name: env
        value: production
  - directive: subnetB
    az: b
    cidr: 10.0.20.0/24
    tags:
      - name: Name
        value: prod-subnet-b
      - name: env
        value: production
  - directive: subnetC
    az: b
    cidr: 10.0.30.0/24
    tags:
      - name: Name
        value: prod-subnet-c
      - name: env
        value: production

Cloudformation template

AWSTemplateFormatVersion: '2010-09-09'
Description: VPC Template
Resources:
  vpc:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: {{ vpc_cidr_block }}
      EnableDnsSupport: true
      EnableDnsHostnames: true
{% if vpc_tags %}
      Tags:
{% for tag in vpc_tags %}
        - Key: {{ tag.name }}
          Value: {{ tag.value }}
{% endfor %}
{% endif %}
{% for subnet in vpc_subnets %}
  {{ subnet.directive }}:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: {{ subnet.az }}
      CidrBlock: {{ subnet.cidr }}
      VpcId: !Ref vpc
{% if subnet.tags %}
      Tags:
{% for tag in subnet.tags %}
        - Key: {{ tag.name }}
          Value: {{ tag.value }}
{% endfor %}
{% endif %}

Example: Security groups

Defining security groups with IP-specific rules is very difficult when you want to keep environment agnosticity and still use the same Cloudformation template for all environments. This however can easily be overcome by providing environment specific array definitions via Jinja2.

Ansible variables

---
# file: staging.yml
# Staging is wiede open, so that developers are able to
# connect from attached VPN's
security_groups:
  - protocol:  tcp
    from_port: 3306
    to_port:   3306
    cidr_ip:   10.0.0.1/32
  - protocol:  tcp
    from_port: 3306
    to_port:   3306
    cidr_ip:   192.168.0.15/32
  - protocol:  tcp
    from_port: 3306
    to_port:   3306
    cidr_ip:   172.16.0.0/16

---
# file: production.yml
# The production environment has far less rules as well as other
# ip ranges.
security_groups:
  - protocol:  tcp
    from_port: 3306
    to_port:   3306
    cidr_ip:   10.0.15.1/32

Cloudformation template

AWSTemplateFormatVersion: '2010-09-09'
Description: VPC Template
Resources:
  rdsSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: RDS security group
{% if security_groups %}
      SecurityGroupIngress:
{% for rule in security_groups %}
        - IpProtocol: "{{ rule.protocol }}"
          FromPort: "{{ rule.from_port }}"
          ToPort: "{{ rule.to_port }}"
          CidrIp: "{{ rule.cidr_ip }}"
{% endfor %}
{% endif %}

Diff

When having enable cloudformation_run_diff, you will be able to see line by line diff output from you local (jinja2 rendered) template against the one which is currently deployed on AWS. To give you an impression about how this looks, see the following example output:

Make sure to run Ansible with --diff to make it work:

$ ansible-playbook play.yml --diff

Json diff

To have it output in json diff mode, set cloudformation_diff_output to json.

TASK [cloudformation : diff cloudformation template file] *********************************************
--- before
+++ after
@@ -38,7 +38,6 @@
             "Type": "AWS::S3::BucketPolicy"
         },
         "s3Bucket": {
-            "DeletionPolicy": "Retain",
             "Properties": {
                 "BucketName": {
                     "Ref": "bucketName"

Yaml diff

To have it output in yaml diff mode, set cloudformation_diff_output to yaml.

TASK [cloudformation : diff cloudformation template file] *********************************************
--- before
+++ after
@@ -14,7 +14,6 @@
               Service: !Sub 'logs.${AWS::Region}.amazonaws.com'
       Bucket: !Ref 's3Bucket'
   s3Bucket:
-    DeletionPolicy: Retain
     Type: AWS::S3::Bucket
     Properties:
       BucketName: !Ref 'bucketName'

Dependencies

This role does not depend on any other roles.

Requirements

Use at least Ansible 2.5 in order to also have --check mode for cloudformation.

The python module cfn_flip is required, when using line-by-line diff of local and remote Cloudformation templates (cloudformation_run_diff=True). This can easily be installed locally:

$ pip install cfn_flip

License

MIT License

Copyright (c) 2017 cytopia