Building Infrastructure with Test Kitchen and Chef

We've selected our favorite tips and tricks created by Michael Crump as well as planned fresh technical content on Azure all April! Miss a day (or more)? Catch up with the series.

Don't have Azure? Grab a free subscription.

When I'm building out infrastructure, it's important to follow the same process as development in testing my code so that I can benefit from consistent, reliable builds. It helps me onboard new engineers into a project and reduces the specialized knowledge individuals need to know to collaborate.

In this post, I'm going to walk through how to set up a project using Chef, Test Kitchen, and the kitchen-azurerm driver. Test Kitchen can be configured to work with a variety of provisioners and drivers, so if you use Puppet or Ansible, for example, you can totally use those instead. In the next post, I'll walk through writing infrastructure tests with InSpec.

First, I need to install the appropriate software on my system. This software is available for a variety of operating systems, so it's not just limited to Mac OS though that's what I'll be using to describe my process. I install the Chef Development Kit(ChefDK) which includes several different pieces of software that are helpful to cookbook development. Cookbooks are the way to encapsulate configuration and policy in Chef.

Sometimes a piece of software that is bundled with the Chef Development kit might have needed updates that aren't available in the latest ChefDK. For example, if I wanted the latest Test Kitchen gem. I follow Steven Murawski's method of updating that gem as needed.

The ChefDK includes Test Kitchen, Chef, and the kitchen-azurerm gem so I'm ready to build infrastructure in the cloud. I can totally do local development with VirtualBox or docker as well.

Next, I need to follow the pre-requisites for the kitchen-azurerm driver as documented in the README.

I wrote a bit about Service Principals here, so I'm going to dive right into setting up a service principal for Test Kitchen as I know I've got the right subscription set up as default.

$ docker run -it microsoft/azure-cli
bash-4.4# az login
bash-4.4# az ad sp create-for-rbac --name sigje_test_kitchen --role Contributor

If you notice the prompt changing above, that's what happens when I go from my local Mac OS terminal prompt into the docker container!

The output of the az ad sp create-for-rbac command will give me back a JSON output with some critical information to populate ~/.azure/credentials.

In ~/.azure/credentials, I set up the file as follows replacing SUBSCRIPTION_ID with my subscription, appID, password, and tenant with the output from the service principal creation.

[SUBSCRIPTION_ID]
client_id = appID
client_secret = password
tenant_id = tenant

Next, I create my cookbook scaffolding using the chef generate command.

$ chef generate cookbook test_software_cookbook

This sets me up with the most basic requirements for a cookbook. I edit the .kitchen.yml file so it looks more like this:

but with my real subscription ID. Any valid machine sizes can be used for the machine_size variable. Additionally, the platform is customizable as well. I took a look at the output of az vm image list --output table and selected the latest Ubuntu Server from Canonical.

I checked to see whether everything is wired up correctly with Test Kitchen, Chef, and Azure.

$ kitchen list

Instance             Driver   Provisioner  Verifier  Transport  Last Action    Last Error
default-ubuntu-1804  Azurerm  ChefZero     Inspec    Ssh

Finally, I can spin up the instance and run any Chef code using kitchen converge. I haven't written any chef code yet, so it's only going to instantiate an instance and install Chef. A successful converge will end with a message something like this:

       Chef Client finished, 0/0 resources updated in 02 seconds
       Downloading files from <default-ubuntu-1804>
       Finished converging <default-ubuntu-1804> (0m43.73s).

From here I can log into the instance using kitchen login. This is handled via the ssh transport and the key that I defined within .kitchen.yml of ~/.ssh/id_kitchen-azurerm. If that key doesn't exist, it will generate it when it's building the instance!

$  kitchen login

Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.18.0-1014-azure x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Mon Apr 22 00:46:55 UTC 2019

  System load:  0.02              Processes:           120
  Usage of /:   4.4% of 28.90GB   Users logged in:     0
  Memory usage: 3%                IP address for eth0: 10.0.0.4
  Swap usage:   0%

 * Ubuntu's Kubernetes 1.14 distributions can bypass Docker and use containerd
   directly, see https://bit.ly/ubuntu-containerd or try it now with

     snap install microk8s --classic

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.


Last login: Mon Apr 22 00:38:37 2019 from 71.193.52.69
azure@vm:~$

This is pretty powerful especially when we can start offloading our "local development" to a node in the cloud that by its very nature will be much closer to our production environment! In the next post, I'll share how to write InSpec tests to test our infrastructure and build on this post.

If I want to clean up my instance, I do kitchen destroy, and that will delete the resource I created.

$ kitchen destroy
-----> Starting Kitchen (v2.0.1)
-----> Destroying <default-ubuntu-1804>...
       Azure environment: Azure
       Destroying Resource Group: kitchen-default-ubuntu-1804-20190418T045457
       Destroy operation accepted and will continue in the background.
       Finished destroying <default-ubuntu-1804> (0m1.24s).

Additional Resources

• Learn about Infrastructure as Code
• Learn Chef on Azure
• Use Chef with Azure
• Install the Chef Extension for Visual Studio Code

We'll be posting articles every day in April, so stay tuned or jump ahead and check out more tips and tricks now.