Provide aws command to create CloudFormation template of resources instead of actual resources

[golharam]

We operate in a controlled environment and all resources need to be created/managed via CloudFormation templates. Also, debugging issues will be easier:

failed to create ComputeEnvironment: error attaching policies to ecsInstanceRole: NoSuchEntity: Policy arn:aws:iam::aws:policy/AmazonEC2ContainerServiceforEC2Role does not exist or is not attachable.

[adamstruck]

I will look into this. Realistically I will likely not have time until the new year.

I am posting an example for my reference: https://github.com/dejonghe/aws-batch-example/blob/master/cloudformation/batch/batch-example.yml

The main custom components that Funnel will need is the JobDef and a custom AMI depending on your use case.

[adamstruck]

Current WIP:

AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS CloudFormation Sample Template Managed Single Batch Job Queue: This
  template demonstrates the usage of simple Job Queue and EC2 style Compute Environment.  **WARNING**
  You will be billed for the AWS resources used if you create a stack from this template.'
Resources:
  ##-------------
  ## Network
  ##-------------
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
  InternetGateway:
    Type: AWS::EC2::InternetGateway
  RouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: VPC
  VPCGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId:
        Ref: VPC
      InternetGatewayId:
        Ref: InternetGateway
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: EC2 Security Group for instances launched in the VPC by Batch
      VpcId:
        Ref: VPC
  Subnet:
    Type: AWS::EC2::Subnet
    Properties:
      CidrBlock: 10.0.0.0/24
      VpcId:
        Ref: VPC
      MapPublicIpOnLaunch: 'True'
  Route:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId:
        Ref: RouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId:
        Ref: InternetGateway
  SubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId:
        Ref: RouteTable
      SubnetId:
        Ref: Subnet

  ##-----------
  ## Roles
  ##-----------
  BatchServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: batch.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole

  EcsInstanceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2008-10-17'
        Statement:
          - Sid: ''
            Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role

  EcsTaskRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2008-10-17'
        Statement:
          - Sid: ''
            Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: FunnelS3
          PolicyDocument:
            Version: '2012-10-17'
            Statement: 
              - Effect: Allow
                Action:
                  - 's3:GetBucketLocation'
                  - 's3:GetObject'
                  - 's3:ListObjects'
                  - 's3:ListBucket'
                  - 's3:CreateBucket'
                  - 's3:PutObject'
                Resource: '*'
        - PolicyName: FunnelDynamodb
          PolicyDocument:
            Version: '2012-10-17'
            Statement: 
              - Effect: Allow
                Action:
                  - 'dynamodb:GetItem'
                  - 'dynamodb:PutItem'
                  - 'dynamodb:UpdateItem'
                  - 'dynamodb:Query'
                Resource: '*'

  ##-----------------
  ## Batch Resources
  ##-----------------
  JobDefinition:
    Type: AWS::Batch::JobDefinition
    Properties:
      Type: container
      ContainerProperties:
        Image: docker.io/ohsu-comp-bio/funnel:latest
        Vcpus: 1
        Memory: 512
        Command:
          - 'worker'
          - 'run'
          - '--EventWriters'
          - 'log'
          - '--EventWriters'
          - 'dynamodb'
          - '--Database'
          - 'dynamodb'
          - '--DynamoDB.Region'
          - Ref: AWS::Region
          - '--DynamoDB.TableBasename'
          - 'funnel'
          - '--Worker.WorkDir'
          - '/opt/funnel-work-dir'
          - '--taskID'
          - 'Ref::TASKID'
        Volumes:
          - Host:
              SourcePath: /var/run/docker.sock
            Name: docker-sock
          - Host:
              SourcePath: /opt/funnel-work-dir
            Name: funnel-work-dir
        MountPoints:
          - ContainerPath: /var/run/docker.sock
            ReadOnly: false
            SourceVolume: docker-sock
          - ContainerPath: /opt/funnel-work-dir
            ReadOnly: false
            SourceVolume: funnel-work-dir
        Environment: []
        Ulimits: []
        ResourceRequirements: []
        User: ""
        ReadonlyRootFilesystem: true
        Privileged: true
        JobRoleArn:
          Ref: EcsTaskRole
      RetryStrategy:
        Attempts: 1

  JobQueue:
    Type: AWS::Batch::JobQueue
    Properties:
      Priority: 1
      ComputeEnvironmentOrder:
        - Order: 1
          ComputeEnvironment:
            Ref: ComputeEnvironment
 
  ComputeEnvironment:
    Type: AWS::Batch::ComputeEnvironment
    Properties:
      Type: MANAGED
      ComputeResources:
        Type: EC2
        MinvCpus: 0
        DesiredvCpus: 0
        MaxvCpus: 256
        InstanceTypes:
          - optimal
        AllocationStrategy: BEST_FIT
        Subnets:
          - Ref: Subnet
        SecurityGroupIds:
          - Ref: SecurityGroup
        InstanceRole:
          Ref: EcsInstanceRole
      ServiceRole:
        Ref: BatchServiceRole

Outputs:
  ComputeEnvironmentArn:
    Value:
      Ref: ComputeEnvironment
  JobQueueArn:
    Value:
      Ref: JobQueue
  JobDefinitionArn:
    Value:
      Ref: JobDefinition

[adamstruck]

@golharam the above seems to produce all of the necessary AWS Batch resources.

Note: the template does not handle setting up the Funnel server.

Let me know if this works for you.

[golharam]

This works. I only needed parts of it, but still good that all is there. I recommend you make a command to output the CloudFormation template and let the user create the resources, rather than having funnel create the resources.