• Challenges Your AWS Cloudformation Skills :AWS: :PROPERTIES: :type: cloudformation :END:

Deploy Jenkins via AWS Cloudformation. Scenario-by-scenario.

[[image-blog:Challenges Your AWS Cloudformation Skills][https://raw.githubusercontent.com/DennyZhang/challenges-cloudformation-jenkins/master/images/cloud-formation.png]]

#+BEGIN_HTML

#+END_HTML

• Blog URL: https://www.dennyzhang.com/challenges-cloudformation-jenkins
• Category: [[https://www.dennyzhang.com/category/AWS/][AWS]]

File me [[https://github.com/DennyZhang/challenges-cloudformation-jenkins/issues][Issues]] or star [[https://github.com/DennyZhang/challenges-cloudformation-jenkins][this repo]].

See more challenges from Denny: [[https://github.com/topics/denny-challenges][#denny-challenges]]

** Scenario-101: Docker Single-Node Jenkins Deployment I

• Objective: Deploy Docker container in AWS
• Requirements: #+BEGIN_EXAMPLE

1. Start an EC2 instance by cloudformation
2. Provision the instance as docker daemon
3. Setup Jenkins container inside the instance #+END_EXAMPLE

• Main Tech: Cloudformation, Docker

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-101.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-101][Scenario-101]]

** Scenario-102: Docker Single-Node Jenkins Deployment II

• Objective: Customize Jenkins docker deployment in AWS
• Requirements: #+BEGIN_EXAMPLE

1. Finish Scenario-101, create a jenkins user by code.
2. Anonymous user can't open the jenkins. Only login user can.
3. When Jenkins is down, get alerts
4. Make sure Jenkins GUI changes can be seamlessly tracked in git repo. #+END_EXAMPLE

• Main Tech: Cloudformation, Docker

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-102.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-102][Scenario-102]]

#+BEGIN_HTML #+END_HTML

** Scenario-201: VM Single-Node Jenkins Deployment I

• Objective: We need a live Jenkins env in public Cloud. Fast and easy.
• Requirements: #+BEGIN_EXAMPLE

1. Use cloudformation to start an EC2 instance
2. Start Jenkins inside the EC2 instance #+END_EXAMPLE

• Main Tech: Cloudformation, Chef

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-201.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-201][Scenario-201]]

#+BEGIN_HTML #+END_HTML

** Scenario-202: VM Single-Node Jenkins Deployment II

• Objective: Customize Jenkins docker deployment in AWS
• Requirements: #+BEGIN_EXAMPLE

1. Finish Scenario-201, create a jenkins user by code.
2. Create a dedicated VPC for the jenkins. And allow selective source IP to access.
3. Anonymous user can't open the jenkins. Only login user can.
4. Make sure Jenkins GUI changes can be seamlessly tracked in git repo. #+END_EXAMPLE

• Main Tech: Cloudformation, Chef, VPC, Slack

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-202.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-202][Scenario-202]]

** Scenario-203: VM Single-Node Jenkins Deployment III

• Objective: Customize Jenkins docker deployment in AWS
• Requirements: #+BEGIN_EXAMPLE

1. Finish Scenario-202
2. Use CF to create a dedicated VPC and start an EC2 #+END_EXAMPLE

• Main Tech: Cloudformation, Chef, VPC, CloudWatch, Slack

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-203.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-203][Scenario-203]]

#+BEGIN_HTML #+END_HTML

** Scenario-301: VM ASG/ELB Jenkins Deployment I

• Objective: Customize Jenkins docker deployment in AWS
• Requirements: #+BEGIN_EXAMPLE

1. Use CF to create ASG and ELB. And monitor ELB
2. Start Jenkins master by ELB. Configure instance count to 1 #+END_EXAMPLE

• Main Tech: Cloudformation, Chef, VPC, CloudWatch, Slack

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-301.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-301][Scenario-301]]

** Scenario-302: VM ASG/ELB Jenkins Deployment II

• Objective: Customize Jenkins docker deployment in AWS
• Requirements: #+BEGIN_EXAMPLE

1. Finish Scenario-301
2. Get slack notificaiton for autoscaling events. Here we assume, one SNS topic has already been created. Verify it by terminating existing instance
3. In ELB, enable monitoring Verify it by terminating existing instance
4. In Jenkins deployment, create a pipeline #+END_EXAMPLE

• Main Tech: Cloudformation, Chef, VPC, CloudWatch, Slack

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-302.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-302][Scenario-302]]

** Scenario-303: VM ASG/ELB Jenkins Deployment II

• Objective: Customize Jenkins docker deployment in AWS
• Requirements: #+BEGIN_EXAMPLE

1. Finish Scenario-302
2. ELB export target group
3. Enable logging for ELB
4. When SNSTopicName is empty, avoid adding SNS notification #+END_EXAMPLE

• Main Tech: Cloudformation, Chef, VPC, CloudWatch, Slack

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-302.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-303][Scenario-303]]

** Scenario-401: VM 2-Nodes Jenkins Deployment I

• Objective: Avoid SPOF by adding 2 Jenkins instance
• Requirements: #+BEGIN_EXAMPLE

1. Start 1 jenkins master and 1 jenkins slave
2. Jenkins master offload request to jenkins slave #+END_EXAMPLE

• Main Tech: Cloudformation, Chef, VPC, CloudWatch, Jenkins Slack Integration, ALB

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-401.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-401][Scenario-401]]
• TODO: how the 2 Jenkins instance gonna to coordinate with each other?

#+BEGIN_HTML #+END_HTML

** Scenario-402: VM 2-Nodes Jenkins Deployment II

• Objective: Jenkins cluster deployment
• Requirements: #+BEGIN_EXAMPLE

1. Start 1 jenkins master and 1 jenkins slave
2. Enable auto-scaling for Jenkins master. With instance count 1
3. Enable auto-scaling for Jenkins slaves. With instance count range from 1 to 3
4. Customized VPC to allow limited network access #+END_EXAMPLE

• Main Tech: Cloudformation, Chef, VPC, CloudWatch, EBS, Jenkins Slack Integration, ALB

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-402.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-402][Scenario-402]]

#+BEGIN_HTML #+END_HTML

** Scenario-501: ECS Jenkins Deployment I

• Objective: Get exposed to docker orchestration service.
• Requirements: #+BEGIN_EXAMPLE

1. Start ECS with 1 node
2. Install a single Jenkins instance #+END_EXAMPLE

• Main Tech: Cloudformation, ECS, EBS

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-501.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-501][Scenario-501]]

** Scenario-502: ECS Jenkins Deployment II

• Objective: Deploy a 2-nodes Jenkins cluster
• Requirements: #+BEGIN_EXAMPLE

1. Start ECS with 2 node
2. Start Jenkins service with 2 instances in ECS. One for master, one for slave.
3. Enable ALB for Jenkins master #+END_EXAMPLE

• Main Tech: Cloudformation, ECS, ELB, CloudWatch, ALB

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-502.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-502][Scenario-502]]
• TODO: How to avoid Jenkins SPOF, theoretically speaking?

#+BEGIN_HTML #+END_HTML

** Scenario-503: ECS Jenkins Deployment III

• Objective: Deploy Jenkins 1 Master 3 Slaves with 2 nodes in ECS2
• Requirements: #+BEGIN_EXAMPLE

1. Start ECS with 2 node
2. Start Jenkins service within ECS. 1 Master and 3 Slaves
3. Enable ALB for Jenkins master #+END_EXAMPLE

• Main Tech: Cloudformation, ECS, ELB, CloudWatch, ALB

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-503.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-503][Scenario-503]]
• TODO: How to avoid Jenkins SPOF, theoretically speaking?

#+BEGIN_HTML #+END_HTML

** Scenario-601: Large Scale Jenkins Deployment I

• Objective: Suppose you have 1K developers to use your Jenkins. Improve scalability, availability, security, etc.
• Requirements: #+BEGIN_EXAMPLE

1. Scalability: multiple Jenkins master instances
2. Availability: Jenkins slave; Jenkins Master
3. Security: VPC, Jenkins authentication integration #+END_EXAMPLE

• Main Tech: Cloudformation, ECS, EBS, ALB

[[https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-jenkins&templateURL=https://s3.amazonaws.com/aws.dennyzhang.com/cf-jenkins-main-601.yml][https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png]]

• See more: [[https://github.com/dennyzhang/challenges-cloudformation-jenkins/tree/master/Scenario-601][Scenario-601]]

#+BEGIN_HTML #+END_HTML ** Highlights

• Highlights For This Case Study #+BEGIN_EXAMPLE

1. OS support: Amazon AMI, Ubuntu and CentOS
2. Use CF to fully automate all deployments
3. Auto-healing: autoscaling group. Improved service availability
4. When issues have happened, we detect it earlier.
5. Pretty secured
6. Quick learner: I have never used CF before. This GitHub repo is the deliverable of 1.5 weeks' part time work. #+END_EXAMPLE

• Discussions & Further Improvements:

TOOD #+BEGIN_EXAMPLE 0. Use docker/ecs/k8s

1. Speed up the whole setup process
2. Reduce the deployment failure rate
3. Improve master HA by using EFS for shared storage of Jenkins HOME
4. Setup 1 master N slaves models
5. Improve error handling. e.g, False positive. when EC2 instance is up, but chef apply hasn't finished
6. Save the cloud bills
7. Use serverless instead of Jenkins hosted solution
8. Use CodeDeploy to replace Chef #+END_EXAMPLE

• Personal Review: #+BEGIN_EXAMPLE

1. Really like container/ECS deployment over VM deployment.
2. Jenkins community don't have HA solution for Jenkins master yet.
3. Jenkins plugins dependency is really troublesome. #+END_EXAMPLE

• Critical Info #+BEGIN_SRC sh

Centos Jenkins Conf

/etc/sysconfig/jenkins

Run chef deployment

chef-solo --config "/home/ec2-user/chef/solo.rb"
--log_level auto -L "/home/ec2-user/log/run_chef_solo.log"
--force-formatter --no-color
--json-attributes "/home/ec2-user/chef/node.json" #+END_SRC

• More Resources License: Code is licensed under [[https://www.dennyzhang.com/wp-content/mit_license.txt][MIT License]].

• Useful links #+BEGIN_EXAMPLE https://github.com/widdix/aws-cf-templates https://github.com/awslabs/startup-kit-templates http://templates.cloudonaut.io/en/stable/jenkins/

Jenkins Security

https://wiki.jenkins.io/display/JENKINS/Standard+Security+Setup https://d0.awsstatic.com/whitepapers/DevOps/Jenkins_on_AWS.pdf #+END_EXAMPLE

#+BEGIN_HTML

#+END_HTML

• org-mode configuration :noexport: #+STARTUP: overview customtime noalign logdone hidestars #+TITLE: Challenges Your AWS Cloudformation Skills #+DESCRIPTION: #+KEYWORDS: #+AUTHOR: Denny Zhang #+EMAIL: [email protected] #+TAGS: noexport(n) #+PRIORITIES: A D C #+OPTIONS: H:3 num:t toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t #+OPTIONS: TeX:t LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc #+EXPORT_EXCLUDE_TAGS: exclude noexport #+SEQ_TODO: TODO HALF ASSIGN | DONE BYPASS DELEGATE CANCELED DEFERRED #+LINK_UP:
  #+LINK_HOME: