Category Archives: python

Dynamic CloudFormation Templates (troposphere and boto3)

CloudFormation Overview

AWS CloudFormation is a really powerful service that enables programmatic creation and modification of AWS resources. It is the centerpiece of most AWS “Infrastructure as Code” implementations enabling developers and operations to achieve idempotence with infrastructure.

Common CloudFormation Challenges

Something any newcomer to CloudFormation will discover when starting to learn the service is the sample templates found online will often be outdated. Most CloudFormation templates that are designed for multi region deployment will utilize mappings of AMIs or instance types. These mappings exist to allow the template to take input from stack parameters to achieve different results based on the dynamic input. The latest AMIs for newer instance types change frequently and not all instance types are available in all regions so mappings are used to allow templates to handle as many scenarios as possible. Amazon Web Services is constantly changing and as a result static templates even with well thought out mappings will quickly go out of date as new AMIs are launched into any of AWS’ many regions. Just take a look at Amazon’s whats new page. The speed of change and new features that Amazon delivers is amazing and writing static CloudFormation templates in JSON or YAML just won’t be able to keep up.

Leveraging Amazon SDKs and other third party tools like troposphere can help organizations make better use of CloudFormation. Today I will share an example of how I can use the Amazon Python SDK (boto3) and troposphere to generate dynamic CloudFormation VPC templates that can be kept up to date as new regions and availability zones are added to EC2.

My need for dynamic CloudFormation templates

Something I often find myself doing to keep up with Amazon Web Services is launching spot instances. It is the cheapest way for me to spin up a short lived instance and install a SDK or a new tool I want to try. I want the ability to have VPCs in any region so I can find the cheapest spot instances available for the instance type I want to launch. Spot prices vary by region and availability zone at any given moment and VPCs without VPN connections are free so it is beneficial for me to have launch options in every AZ of every region.

The VPCs I need in each region are fairly simple the requirements are:

  • An internet gateway attached to the VPC
  • A subnet created in each availability zone in the region
  • Network ACLs to allow connectivity to instances in the subnets
  • Routing from the subnets to the internet gateway for external connectivity
  • Public IPs assigned to all launched instances by default

The below python script will generate a CloudFormation template that can be used to create a VPC in any available region, using all accessible availability zones in the region. The list of regions and availability zones is queried directly from the AWS APIs. If tomorrow a new availability zone becomes available in any of AWS’s many regions, I can run the template generation script to generate an updated version of the CloudFormation template. The updated template can then be used to update the CloudFormation stack to immediately make use of the new availability zone. The same concept applies to new regions if AWS launches a new region tomorrow I will be able to make use of it immediately without writing any new code.

Walkthrough of using this script to generate a CloudFormation template

Basic Requirements for this script:

  • Python3 installed and configured
  • boto3 installed and ec2 describe permissions configured
  • troposphere installed

With the above requirements met I can execute the python script to generate a CloudFormation template. The script will prompt for the desired region which in this example I will enter as us-east-1. If an invalid or unavailable region is entered, the script will display a list of valid regions to be used. The script then displays the path to the CloudFormation template that is generated to the current working directory.

If you enjoy reading JSON files you can take a peek at the generated template file.

Launching the CloudFormation template

With the template generated I can now head to the CloudFormation console in us-east-1 to launch my spot instance VPC stack. It is also possible to use the Amazon SDK’s to launch this CloudFormation template for true Infrastructure as Code but for this example I will be using the console. From the CloudFormation console I click the Create Stack button.

On the next screen I choose “Upload a template to Amazon S3” and click the browse button. Using the file upload dialog popup I can navigate to the file that was displayed from the script execution above and click Open. The file will be automatically uploaded to an S3 bucket for me and its ready to be consumed by CloudFormation to create the stack.

After clicking next I am prompted to input a stack name, lets go with VirginiaSpotInstanceVPC.

The next screen displays some stack launch options, I am taking the default so I omitted the screenshot. The final step of the wizard is a review page. Everything looks good so I am ready to click create and watch CloudFormation works its magic.

CloudFormation Stack Creation

Once the stack is launched the console will exit the Create Stack wizard and return me to the CloudFormation console where the stack launch status is displayed. At this point the stack creation is still in progress.

After a minute or so I can utilize the refresh button on the top right of the CloudFormation console to see if the stack creation is done. The events tab can be used to view time stamped logs of the resource types the stack altered.

The status is now green / CREATE_COMPLETE so I can go check out my shiny new spot instance VPC in the VPC console of the us-east-1 region.

On the left hand side I use the filter dropdown to only display objects that apply to the new spot instance VPC and check on the subnets to make sure I have a subnet in each availability zone in the region. Indeed I do.

Summary

I was able to identify my need for dynamic infrastructure creation and leverage python, boto3, troposphere, and CloudFormation to create templates that should be reusable and update-able for the foreseeable future. The open source troposphere library opens up the possibility of using loops to iterate over returned objects from real-time calls to the AWS APIs with the python AWS SDK. Another benefit of using troposphere is I don’t have to sweat JSON or YAML syntax, I can use my working knowledge of python syntax instead.

Of course nothing is totally future proof in the fast moving field of cloud computing but I think this approach sure beats writing straight JSON or YAML templates and then updating them everytime a new region or AZ is added to AWS.

That wraps it up for this example of using boto3 and troposphere to help manage infrastructure as code.

Below are some of the documentation links I used to create this solution.

Fun with Google’s programming challenge site (foo.bar)

A few weeks ago I was browsing some threads discussing reddit’s april fool’s day /r/place experiment. I was interested in seeing some of the overlay scripts people whipped up on short notice and found myself in a script discussion thread.  In this thread I came across a redditor giving others a heads up that if you are ever searching for python programming terms and google asks you if you are up for a challenge, say yes, because it is a job interview. I thought that was really interesting so I did some research and came across a few older hacker news threads discussing this practice and this article that confirmed this thing existed.

As I am just venturing down the road of learning python (I started learning less than a month ago), I find myself googling almost everything related to the code I am writing. If this thing is real I am probably going to see it eventually…

Fast forward to today and I am working on writing an AWS lambda/boto function in python and needed a reminder on list comprehension syntax. I went to google and searched python list comprehension. As I snapped my cursor to click the first link, I saw the page morph. I was instantly reminded of the foobar challenge and knew I had just been invited. Unfortunately I had clicked too fast and left the search results page. I went back and refreshed and re-searched but the invite was gone!

I went to lunch and pondered over what a fun learning opportunity I missed. When I got back to my desk, I went back to google to retrace my steps, hopeful the invites were google account linked and perhaps I would get another invite. I never technically “declined” my invite I just missed it. Sure enough, one google search later, I had my invite. This is what it looked like.

After accepting the invite I was taken to google’s foobar challenge site. It was a web app with a very minimalist feel. I was greeted with what looked like a linux command prompt so I typed “ls”. It listed a bunch of files, one called journal.txt and another called start_here.txt. Using “cat” on these files start_here.txt gave me instructions on how to request a challenge and journal.txt set the stage for challenges plot line.

LEVEL 1
=======

Success! You’ve managed to infiltrate Commander Lambda’s evil organization, and finally earned yourself an entry-level position as a Minion on her space station. From here, you just might be able to subvert her plans to use the LAMBCHOP doomsday device to destroy Bunny Planet. Problem is, Minions are the lowest of the low in the Lambda hierarchy. Better buck up and get working, or you’ll never make it to the top…

Next time Bunny HQ needs someone to infiltrate a space station to rescue prisoners, youre going to tell them to make sure stay up for 48 hours straight scrubbing toilets is part of the job description. Its only fair to warn people, after all.

I followed the instructions and requested a challenge. I was given “the cake is not a lie!” challenge as my level 1 challenge. A timer appeared on the lower left hand side of the screen, indicating I had 48 hours to solve this challenge. A new directory was created in my home directory of this virtual terminal. Here is what the challenge looked like.

The cake is not a lie!
======================

Commander Lambda has had an incredibly successful week: she completed the first test run of her LAMBCHOP doomsday device, she captured six key members of the Bunny Rebellion, and she beat her personal high score in Tetris. To celebrate, she’s ordered cake for everyone – even the lowliest of minions! But competition among minions is fierce, and if you don’t cut exactly equal slices of cake for everyone, you’ll get in big trouble.

The cake is round, and decorated with M&Ms in a circle around the edge. But while the rest of the cake is uniform, the M&Ms are not: there are multiple colors, and every minion must get exactly the same sequence of M&Ms. Commander Lambda hates waste and will not tolerate any leftovers, so you also want to make sure you can serve the entire cake.

To help you best cut the cake, you have turned the sequence of colors of the M&Ms on the cake into a string: each possible letter (between a and z) corresponds to a unique color, and the sequence of M&Ms is given clockwise (the decorations form a circle around the outer edge of the cake).

Write a function called answer(s) that, given a non-empty string less than 200 characters in length describing the sequence of M&Ms, returns the maximum number of equal parts that can be cut from the cake without leaving any leftovers.

Languages
=========

To provide a Python solution, edit solution.py
To provide a Java solution, edit solution.java

Test cases
==========

Inputs:
(string) s = “abccbaabccba”
Output:
(int) 2

Inputs:
(string) s = “abcabcabcabc”
Output:
(int) 4

Use verify [file] to test your solution and see how it does. When you are finished editing your code, use submit [file] to submit your answer. If your solution passes the test cases, it will be removed from your home folder.

I read over this challenge and knew this was no task for work. This was going to be a brain twister for a python novice such as myself and I’m not being paid to play python games afterall. My immediate schedule is quite busy and I didn’t have a ton of time to spend on this, so I decided to think on this challenge on my commute home.

I thought about the intricacies of this problem while dealing with my daily dose of New Jersey’s rush hour. The function would need to return 0 if the cake could not be cut without leftovers or if the string was empty. I needed to identify possible M&M patterns from the input string. The pattern surely could not be longer than half the length of the string. I needed to check to see if there were remainders  (probably with a modulus) when chopping the string with the possible patterns and discard those results. I had to wrap the string somehow because the function needed to iterate over all possible starting points of the string to find valid solutions. There would be edge cases / test cases on that part I was sure due to the way it was stated on the description that the M&Ms were laid out clockwise. As such a python beginner, who had to google list comprehension syntax to get here, I knew I wasn’t solving this in the few hours of free time I had in the next two days.

So I did what anyone in IT does when they don’t know a solution off the top of their head, I went to the search engines to discover some discussions on this problem. A quick search later and I found this stackoverflow question with a valid java answer by Jawad Le Wywadi. Further searches for a python solution turned up blank. In the end, this blog post may very well change that.

Here is the example code from the stack overflow question written in java:

Well, with a possible answer in hand, I still wanted to at least have somewhat of a challenge so I decided to see if I could turn this solution into a valid python solution to the problem. There was a text file with constraints listed, so I would need to take those into consideration. Notably for me was the fact that this needed to run on python 2.7.6

Java
====

Your code will be compiled using standard Java 7. It must implement the answer() method in the solution stub.

Execution time is limited. Some classes are restricted (e.g. java.lang.ClassLoader). You will see a notice if you use a restricted class when you verify your solution.

Third-party libraries, input/output operations, spawning threads or processes and changes to the execution environment are not allowed.

Python
======

Your code will run inside a Python 2.7.6 sandbox.

Standard libraries are supported except for bz2, crypt, fcntl, mmap, pwd, pyexpat, select, signal, termios, thread, time, unicodedata, zipimport, zlib.

After a few more searches and some quick print debugging, I was getting closer. While I know I had zero chance of coming up with this solution on my own with my current time constraints, I was still able to learn a bit of python syntax from this challenge when converting the java code into a python solution. The python code was solving all the inputs I threw at it despite being overly verbose and unoptimized. Here is that code

I used the foobar editor to put my code into solution.py and ran the verify command to see if I had a solution.

Well look at that, All test cases passed! I had indeed googled my solution to google’s challenge.

After verifying my solution, I submitted it to move on to level 2, and was greeted by a jumping ASCII bunny on the terminal. Who doesn’t love ASCII art?

For this next challenge level, I am going to wait until I have a free weekend and proper time to possibly come up with my own solution.

Configure Visual Studio Code to run Python on Windows

Recently I was looking at my IDE options for writing some python applications. I really want to get more familiar with Visual Studio Code so I picked that to begin learning on. My only requirement was to pick an IDE I could use on both Windows and Linux. I know that many other IDEs are more polished and would be easier to configure but I wanted to challenge myself a little.

After I downloaded the latest version of python, I installed python via the executable installer. During the install there is a check box you can select to add python to your path, so I chose that option.

Next up I downloaded Visual Studio Code and also used the executable installer. Again there is an option to add Visual Studio Code to the path environment variable, so I chose that option as well. After a reboot my path looked like this.

You can see the paths to python and Visual Studio Code are present. This should make working with these new tools a bit easier. To make sure I could run python from the shell, I went ahead and launched python from the PowerShell console and outputted some Hello World.

With shell execution confirmed it was time to move on to the IDE configuration. I opened Visual Studio Code and installed the python extension. Then I opened a folder I created in My Documents for python scripts within Visual Studio Code. I had already picked up that opening up a folder in VS code was a first step to code execution from some earlier learning on Ubuntu with PowerShell and Dot NET Core. Inside that new folder, I created HelloWorld.py with a simple print Hello World line. When I went to run the HelloWorld.py with Control+Shift+B, I hit a wall. No task runner configured error. That is where I discovered I needed to configure the task runner to be able to execute python from Visual Studio Code. To configure the task runner, open the command palette either through the Menus via View -> Command Palette or use the keyboard shortcut Control+Shift+P and type Task Configure, to find the configure task runner selection. Then Choose Other, and make your tasks.json look like this:

All I needed to change was the “command” line to python and the “args” line to [“${file}”], once that was done I was able to execute python from Visual Studio Code as seen below.

Credit to codingisforyou on youtube for making a video covering this as well as diving a bit deeper into the debugging configuration of VS code. In his video he needed to include the full path to the python executable including escaping some of the backslash characters, which I assume is due to python not being in his environment variables.

Now my python environment inside Visual Studio Code is all configured and actually looks like the Microsoft Documentation.