Creating your cloud servers with Terraform

Sometimes when you handle a lot of servers in the cloud, it is pretty easy to get lost on your infrastructure. “Where is that freaking server that I can’t find?”, or even “Why is this instance for?”.

In this post, I will introduce one tool that I found myself liking a lot: To have a bootstrap of an Infrastructure as Code flow in my applications. Terraform. Infrastructure as Code (IaC) allows us to have a repository with code that describes our infrastructure. This way, we can avoid reminding how rebuild the entire infrastructure for an application. It will be on the code and it can be versioned and tested. And if something goes wrong, we can revert it. It is very useful having a continuous integration of the infrastructure. The whole team knows how it was built and all the pieces. You can apply the same flow that you use in you app code to your infrastructure, i.e: Someone makes a change, open a Pull Request, someone reviews it and after it is approved, you merge and your CI tools apply the changes in your environment.

Why Terraform? What is the difference between Chef, Puppet or Ansible?

Chef, Puppet and Ansible are IaC tools too, but they focus on configuring operating system and applications. They are called Configuration Management Tools and they also can build infrastructure on the cloud with a help of plugins, but usually it is hard to configure and sometimes it is limited. With Terraform you can build from the services to the networking part. You can use Terraform to create the infrastructure and a configuration management tool to configure the applications. Terraform can’t replace your configuration management tool, but it’s made to work together with it.

A practical example

In order to follow this article, you’ll need an AWS account.

Let’s create the servers for our web application. Take a look at this diagram of the resources that we’ll build.

We will create 2 subnets (one for public access and another private). We have Elastic Load Balancer in the public subnet to handle the traffic to our web servers. Our web servers will be on the private subnet and it will only be accessible through the Load Balancer. This mean that we won’t have direct access to make connections (for example, SSH) on the server. In order to access via SSH an instance on a private subnet, you’ll need a bastion host and connect to the web server through it. Thus, we will create the bastion host on the public subnet.

Before getting your hands dirty, you need to install Terraform. Follow the instructions of this link to install it in your machine.

Directory structure

Let’s create a folder to handle our infrastructure code.

mkdir ~/terraform

I like to follow this pattern when working with Terraform:

├── modules
│ ├── networking
│ └── web
├── production
└── staging

Let’s create this folder structure

cd ~/terraform
mkdir -p modules/{networking,web} production staging

Our modules folder contains all the shared code to create the pieces of the infrastructure (web servers, app servers, databases, vpc, etc). Each folder inside the modules folder is related to a specific module.

Next, I have folders for my environments (staging, production, qa, development, etc). Each of this folder contains code to use our shared modules and create a different architecture for each environment (This is my personal approach using Terraform, but feel free to work on a different way).

Our first module: Networking

Let’s create our networking module. This will be responsible for creating the networking pieces of our infrastructure, like VPC, subnets, routing table, NAT server and the bastion instance.

Before we get deep in the code, I wanna explain how terraform works:

Terraform will provide us with some commands. Some of them are:

plan : Displays all the changes that Terraform makes on our infrastructure

apply : Executes all the changes to the infrastructure

destroy : Destroys everything that was created with Terraform

When you run Terraform inside a directory, it loads ALL .tf files from the directory and execute them (will not load on subfolders). Terraform will first create a graph of the resources to apply only in the final phase, so you don’t need to specify the resources in any specific order. The graph will determine the relations between the resources and ensure that Terraform creates they in the right order.

Continuing on our networking module

Enter in the networking module

cd modules/networking

Let’s create our first tf file. The one that specifies all variables needed for our module.

touch variables.tf

Insert the following content on the variables.tf file:

These are all variables that our networking module needs in order to create all resources. We need the CIDR for the VPC and the subnets, the AWS region that we will use, the key name and the environment that we are building.

This is the way that you specify a variable in Terraform

variable "variable\_name" {
  description = "The description of the variable"
  default = "A default value if this isn't set
}

Ok. Now we have our variables.tf file to specify the interface of our module. Let’s create the file that will create networking stuffs for our module. Create the main.tf file in the networking folder. (you can specify any name that you want. Remember, all tf files will be loaded).

touch main.tf

Insert the following content on the main.tf file:

Here we are creating all the networking part of our infrastructure based on the diagram that we saw. A VPC, both subnets (public and private), the Internet Gateway to the public subnet, the NAT server for the private subnet, the bastion host and all security group for the VPC, allowing inbound and outbound inside the VPC, and the security group for the bastion host, allowing the SSH on the Port 22. You can check this link from AWS for more details.

This is how we create a resource on Terraform.

resource "resource" "name" {
  attribute = "value"
}

The resource is the name of the resource that we want to build. Each cloud provider has different resources. For example, these resources from AWS. We can also concatenate values. Remember that we created the variables file? We use them here with the var.variable_name . Like in this part of the code, which we use the key_name variable that we specified in the variables file:

resource "aws\_instance" "bastion" {
  ...
  key\_name = "${var.key\_name}"
  ...
}

This is how we created the instance and called its bastion. You can also get property of this resource in other parts of the code. Example:

resource "someresource" "somename" {
  attribute = "${aws\_instance.bastion.id}"
}

We use the same idea of var concatenation. But we specify ${resource_type.resource_name.property} .

Our networking module is almost ready. We need to output some variables after the module build the resources, so we can use it in other parts of the code. Terraform has the output command, allowing us to expose variables.

Create the output.tf file inside the networking folder.

touch output.tf

Insert the following content on the output.tf file:

This is how we output a variable from our module:

output "variable\_name" {
  value = "variable value"
}

This will allow us to get these variables outside the module.

Using our module to build the networking from our environment

Now that our networking module is ready, we can use it to build our networking from our environment (i.e, staging).

This is how our terraform folder looks now:

├── modules
│ ├── networking
│ │ ├── main.tf
│ │ ├── output.tf
│ │ └── variables.tf
│ └── web
├── production
└── staging

Go into staging folder.

cd ~/terraform/staging

First, create the public key for the staging servers

ssh-keygen -t rsa -C "staging\_key" -f ./staging\_key

Let’s create our main file. In this file, we will specify information of the AWS provider.

touch \_main.tf

(We use _ at the beginning of the name because since terraform loads all files alphabetically, we need this to be loaded first, since it will create the keypair)

You can specify things like Access and secret key in some ways:

• Specify it directly in the provider (not recommended)

provider "aws" {
  region = "us-west-1"
  access\_key = "myaccesskey"
  secret\_key = "mysecretkey"
}

• Using the AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables

$ export AWS\_ACCESS\_KEY\_ID="myaccesskey"
$ export AWS\_SECRET\_ACCESS\_KEY="mysecretkey"

terraform plan

The second option is recommended because you don’t need to expose your secrets on the file. Bonus point : If you have the AWS cli you don’t need to export these variables. Only run the aws configure command and terraform will use the variables that you set on it.

Insert the following content on the _main.tf file:

We will only specify the region to the provider. Both access and secret key, we will rely on the AWS cli .

Now create the networking.tf file. It will use our module to create the resources.

touch networking.tf

Insert the following content on the networking.tf file:

This is how we use modules on Terraform.

module "name" {
  source = "location\_path"
  attribute = "value"
}

The module attributes are all variables that we specified before in the variables.tf file from our networking module. Look that we are passing more variables to our module attributes. We need our environment to require these variables too. Create the variables.tf file for our staging environment.

touch variables.tf

Add the following content to the variables.tf file:

This file follows the same pattern of the module’s variables file. These are all variables that we need to build our staging networking piece.

Ok… I think that we are ready to go. Your terraform’s folder structure should be like this:

├── modules
│ ├── networking
│ │ ├── main.tf
│ │ ├── output.tf
│ │ └── variables.tf
│ └── web
├── production
└── staging
    ├── \_main.tf
    ├── networking.tf
    └── variables.tf

Run this command on staging folder: (Terraform’s commands should be run on the environments folder).

cd ~/terraform/staging
terraform get
terraform plan

*the terraform get command only syncs all modules.

After executing the terraform plan command, it will ask you a lot of informations (our variables). Answer they:

It will output a lot of things. Resources that will be created. You can analyze it to check if everything is ok. terraform plan will output only the planned change of the infrastructure.

Now, let’s apply these modifications.

terraform apply

But it is asking for all that information again. To avoid this, you can follow 2 ways to automatically inject variable’s value to Terraform

• Using a terraform.tfvars file and remember to ignore this file in your VCS.
• Specifying variables in Environment variables. TF_VAR_environment=staging terraform apply (this can be useful when you run through some CI tool)

We will follow the first way. So, create a terraform.tfvars file

touch terraform.tfvars

Insert the following content on terraform.tfvars :

Now you can run the apply command without being asked for variable’s value.

terraform apply

You should receive the following message:

Congratulations..

This will generate 2 files on the staging folder: terraform.tfstate and terraform.tfstate.backup

Terraform controls all their resources on this terraform.tfstate file. You should NEVER delete this file. If you do, terraform will think that it needs to create new resources and will lose tracking with the others that it has been already created.

And how can I make my team in sync with the state?

You have 2 ways to keep the team with the remote in sync. You can commit this .tfstate file to your VCS repository, or use Terraform Remote State. If you rely on terraform’s remote state, I really recommend you to use some wrapper tool for Terraform like Terragrunt. It can handle the remote state locking and initializing it for you.

Creating the web servers

This is our folder structure until now.

├── modules
│ ├── networking
│ │ ├── main.tf
│ │ ├── output.tf
│ │ └── variables.tf
│ └── web
├── production
└── staging
    ├── \_main.tf
    ├── networking.tf
    ├── staging\_key
    ├── staging\_key.pub
    ├── terraform.tfstate
    ├── terraform.tfstate.backup
    ├── terraform.tfvars
    └── variables.tf

Go into our web module

cd ~/terrform/modules/web

Following the same flow we used to create the networking module, let’s create our variables file to specify the interface to our module:

touch variables.tf

Insert the following content on variables.tf :

Create the main.tf file to handle the creation of the resources.

touch main.tf

Insert the following content on main.tf :

This file is practically the same from our networking’s main.tf, we are only creating different resources.

Some differences that you can note:

• In this example we are using the count attribute. It specifies to Terraform, to create N times this resource. If we pass 5 on the value, it will create 5 instances.

resource "aws\_instance" "web" {
  count = "${var.web\_instance\_count}"
  ...
}

and you can create dynamic names with the counting number using the count.index property, for example:

tags {
  Name = "web-server-${count.index + 1}"
}

• We are passing a file to the user_data property. In order to execute some code on the instance initialization, we need to pass the user_data attribute to our instance and we are specifying a file.

user\_data = "${file("${path.module}/files/user\_data.sh")}"

This will load the content of the user_data.sh file and pass to the attribute.

But we haven’t created this file yet, let’s do it. On the web module folder, create the files folder and the user_data.sh file.

mkdir files
touch files/user\_data.sh

Insert the following content on the files/user_data.sh :

This will install the nginx when the instance is created.

Now, let’s create the output.tf file to get the load balancer’s DNS after the execution.

touch output.tf

Insert the following content on the output.tf :

Our web module is done.

This is how our terraform structure is now:

├── modules
│ ├── networking
│ │ ├── main.tf
│ │ ├── output.tf
│ │ └── variables.tf
│ └── web
│ ├── files
│ │ └── user\_data.sh
│ ├── main.tf
│ ├── output.tf
│ └── variables.tf
├── production
└── staging
    ├── \_main.tf
    ├── networking.tf
    ├── staging\_key
    ├── staging\_key.pub
    ├── terraform.tfstate
    ├── terraform.tfstate.backup
    ├── terraform.tfvars
    └── variables.tf

Using our web module

Let’s back to our staging folder

cd ~/terraform
cd staging

Now, we will use our recent created web module. Create a web.tf file

touch web.tf

Insert the following content on the web.tf :

This is pretty much the same way we used in our networking module. We are using almost the same variables that we already specified (except for web_instance_count . And some variables, we pass the output from our networking module.

"${module.networking.public\_subnet\_id}"

This way we get the public_subnet_id output created on the networking module.

Let’s add the web_instance_count variable to our variables.tf file and terraform.tfvars . This variable represents the number of web instances that we will be created.

Your variables.tf from staging folder should be like this:

And your terraform.tfvars should be like this:

Let’s create an output.tf for our staging environment. With this, we can get the ELB hostname from our web module.

touch output.tf

Add the following content to output.tf :

This is the final directory structure that we have:

├── modules
│ ├── networking
│ │ ├── main.tf
│ │ ├── output.tf
│ │ └── variables.tf
│ └── web
│ ├── files
│ │ └── user\_data.sh
│ ├── main.tf
│ ├── output.tf
│ └── variables.tf
├── production
└── staging
    ├── \_main.tf
    ├── networking.tf
    ├── output.tf
    ├── staging\_key
    ├── staging\_key.pub
    ├── terraform.tfstate
    ├── terraform.tfstate.backup
    ├── terraform.tfvars
    ├── variables.tf
    └── web.tf

You can now run terraform plan to check which resources will be created. (before, run terraform get to update the modules)

terraform get
terraform plan

Done. Terraform will create 5 new resources from our web module.

Let’s apply it.

terraform apply

Yay.. our instances was created. Let’s get our Load Balancer DNS and try to open it on a Browser:

[terraform](http://%24(terraform) output elb\_hostname

This command will return the DNS from our Load Balancer. Open it on a browser:

It’s working!!! With this, you finished the creation of your infrastructure.

Final Notes:

Your web instances don’t have public ips:

In order to SSH then, you need to use the bastion host created on the networking module.

Get the bastion host public IP,

and SSH on it with the -A flag to enable agent forwarding:

chmod 400 staging\_key.pub

ssh-add -K staging\_key

ssh -A ubuntu@52.53.227.241

Now, inside the Bastion host, you can connect into your web server private IP:

ssh ubuntu@10.0.2.113

Now, destroy everything

terraform destroy

You can get the full code of the example here.

That’s all.

Cheers,

🍻