CloudFormation Overview
AWS CloudFormation is a really powerful service that enables programmatic creation and modification of AWS resources. It is the centerpiece of most AWS “Infrastructure as Code” implementations enabling developers and operations to achieve idempotence with infrastructure.
Common CloudFormation Challenges
Something any newcomer to CloudFormation will discover when starting to learn the service is the sample templates found online will often be outdated. Most CloudFormation templates that are designed for multi region deployment will utilize mappings of AMIs or instance types. These mappings exist to allow the template to take input from stack parameters to achieve different results based on the dynamic input. The latest AMIs for newer instance types change frequently and not all instance types are available in all regions so mappings are used to allow templates to handle as many scenarios as possible. Amazon Web Services is constantly changing and as a result static templates even with well thought out mappings will quickly go out of date as new AMIs are launched into any of AWS’ many regions. Just take a look at Amazon’s whats new page. The speed of change and new features that Amazon delivers is amazing and writing static CloudFormation templates in JSON or YAML just won’t be able to keep up.
Leveraging Amazon SDKs and other third party tools like troposphere can help organizations make better use of CloudFormation. Today I will share an example of how I can use the Amazon Python SDK (boto3) and troposphere to generate dynamic CloudFormation VPC templates that can be kept up to date as new regions and availability zones are added to EC2.
My need for dynamic CloudFormation templates
Something I often find myself doing to keep up with Amazon Web Services is launching spot instances. It is the cheapest way for me to spin up a short lived instance and install a SDK or a new tool I want to try. I want the ability to have VPCs in any region so I can find the cheapest spot instances available for the instance type I want to launch. Spot prices vary by region and availability zone at any given moment and VPCs without VPN connections are free so it is beneficial for me to have launch options in every AZ of every region.
The VPCs I need in each region are fairly simple the requirements are:
- An internet gateway attached to the VPC
- A subnet created in each availability zone in the region
- Network ACLs to allow connectivity to instances in the subnets
- Routing from the subnets to the internet gateway for external connectivity
- Public IPs assigned to all launched instances by default
The below python script will generate a CloudFormation template that can be used to create a VPC in any available region, using all accessible availability zones in the region. The list of regions and availability zones is queried directly from the AWS APIs. If tomorrow a new availability zone becomes available in any of AWS’s many regions, I can run the template generation script to generate an updated version of the CloudFormation template. The updated template can then be used to update the CloudFormation stack to immediately make use of the new availability zone. The same concept applies to new regions if AWS launches a new region tomorrow I will be able to make use of it immediately without writing any new code.
Walkthrough of using this script to generate a CloudFormation template
Basic Requirements for this script:
- Python3 installed and configured
- boto3 installed and ec2 describe permissions configured
- troposphere installed
With the above requirements met I can execute the python script to generate a CloudFormation template. The script will prompt for the desired region which in this example I will enter as us-east-1. If an invalid or unavailable region is entered, the script will display a list of valid regions to be used. The script then displays the path to the CloudFormation template that is generated to the current working directory.
If you enjoy reading JSON files you can take a peek at the generated template file.
Launching the CloudFormation template
With the template generated I can now head to the CloudFormation console in us-east-1 to launch my spot instance VPC stack. It is also possible to use the Amazon SDK’s to launch this CloudFormation template for true Infrastructure as Code but for this example I will be using the console. From the CloudFormation console I click the Create Stack button.
On the next screen I choose “Upload a template to Amazon S3” and click the browse button. Using the file upload dialog popup I can navigate to the file that was displayed from the script execution above and click Open. The file will be automatically uploaded to an S3 bucket for me and its ready to be consumed by CloudFormation to create the stack.
After clicking next I am prompted to input a stack name, lets go with VirginiaSpotInstanceVPC.
The next screen displays some stack launch options, I am taking the default so I omitted the screenshot. The final step of the wizard is a review page. Everything looks good so I am ready to click create and watch CloudFormation works its magic.
CloudFormation Stack Creation
Once the stack is launched the console will exit the Create Stack wizard and return me to the CloudFormation console where the stack launch status is displayed. At this point the stack creation is still in progress.
After a minute or so I can utilize the refresh button on the top right of the CloudFormation console to see if the stack creation is done. The events tab can be used to view time stamped logs of the resource types the stack altered.
The status is now green / CREATE_COMPLETE so I can go check out my shiny new spot instance VPC in the VPC console of the us-east-1 region.
On the left hand side I use the filter dropdown to only display objects that apply to the new spot instance VPC and check on the subnets to make sure I have a subnet in each availability zone in the region. Indeed I do.
Summary
I was able to identify my need for dynamic infrastructure creation and leverage python, boto3, troposphere, and CloudFormation to create templates that should be reusable and update-able for the foreseeable future. The open source troposphere library opens up the possibility of using loops to iterate over returned objects from real-time calls to the AWS APIs with the python AWS SDK. Another benefit of using troposphere is I don’t have to sweat JSON or YAML syntax, I can use my working knowledge of python syntax instead.
Of course nothing is totally future proof in the fast moving field of cloud computing but I think this approach sure beats writing straight JSON or YAML templates and then updating them everytime a new region or AZ is added to AWS.
That wraps it up for this example of using boto3 and troposphere to help manage infrastructure as code.
Below are some of the documentation links I used to create this solution.
