Terraform Certification (Part 13): Use Terraform outside the core workflow

We went through the Terraform core workflow in an earlier lesson. This workflow is what you most often use. But sometimes you must do more than what this workflow can offer. This might include things like importing existing resources into your Terraform configuration or manipulating your Terraform state in some way. These issues belong to outside the core workflow.

In this lesson I will cover part 4 of the Certified Terraform Associate exam curriculum. This part is outlined below:

Importing existing infrastructure to Terraform

Usually it is the case that you start out with a number of resources that you have created through some other means than Terraform. This could be through ClickOps¹ or perhaps using a command line tool. Then you come to a point where you would like to include these resources in Terraform. Your alternatives are:

1. Delete the resource you created manually, then re-create the resource using Terraform. You have to consider potential downtime in your application using this approach, depending on what type of resource it is.
2. Import the resource into your Terraform configuration using terraform import.

Let's see how the second alternative works. In the following scenario I have created an Azure storage account using the Azure portal:

The important details are highlighted in the figure and explained below:

• The name of the storage account is importthisaccount.
• The storage account is placed in a resource group named rg-terraform-course.
• The location for the storage account is West Europe.
• The storage account is configured with Standard performance, LRS replication, and it is a general purpose V2 account kind.

I will write a small Terraform configuration that will be our starting point:

// main.tf

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
  }
}

// configure the azurerm provider
provider "azurerm" {
  features {}
}

// the resource we would like to import
resource "azurerm_storage_account" "my_storage_account" {}

Now we have a Terraform configuration where our storage account is included. However, I have left the resource block completely empty. Next I initialize the Terraform configuration using terraform init:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/azurerm...
- Installing hashicorp/azurerm v3.42.0...
- Installed hashicorp/azurerm v3.42.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

To be able to import a resource we must know what ID it has. For resources in Azure the ID is simply the corresponding Azure resource ID. Simple! A resource ID in Azure has the format:

/subscriptions/<id>/resourceGroups/<name>/providers/<provider>/<resource type>/<resource name>

The details of what a resource ID looks like for the provider you use vary from provider to provider, so you need to see the documentation for your provider. Using a properly formatted ID together with the terraform import command we are now ready to import the storage account into our Terraform state:

$ terraform import azurerm_storage_account.my_storage_account <resource id>

azurerm_storage_account.my_storage_account: Importing from ID "<azure resource id>"...
azurerm_storage_account.my_storage_account: Import prepared!
  Prepared azurerm_storage_account for import
azurerm_storage_account.my_storage_account: Refreshing state... [id=<resource id>]

Import successful!

I have masked the resource ID in the above command for brevity. Apart from the resource ID the command also takes a resource spec address to where the resource should be imported. The resource spec address has the format <resource type>.<local name>². To confirm that our resource has been imported into our state we can run terraform show:

$ terraform show

# azurerm_storage_account.my_storage_account:
resource "azurerm_storage_account" "my_storage_account" {
    access_tier                       = "Hot"
    account_kind                      = "StorageV2"
    account_replication_type          = "LRS"
    account_tier                      = "Standard"
    ...
}

The terraform show command shows the resources in our current state. Before we can use Terraform to manage our storage account we must also fill in the blanks in the configuration. How you do this is up to you, you could either copy everything from the state file into main.tf or you can just include the required arguments by hand. I used this last approach and ended up with this configuration:

// main.tf

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
  }
}

// configure the azurerm provider
provider "azurerm" {
  features {}
}

// the resource we would like to import
resource "azurerm_storage_account" "my_storage_account" {
  name                     = "importthisaccount"
  resource_group_name      = "rg-terraform-course"
  location                 = "West Europe"
  account_tier             = "Standard"
  account_replication_type = "LRS"
  account_kind             = "StorageV2"
}

If I now run terraform plan I can see that Terraform knows about my storage account and knows there are no changes to apply:

$ terraform plan

azurerm_storage_account.my_storage_account: Refreshing state... [id=<resource id>]

No changes. Your infrastructure matches the configuration.

We have successfully imported a resource into Terraform!

Interacting with the Terraform state using the CLI

The Terraform CLI includes a subcommand named state that allows you to interact with the Terraform state. These commands are for advanced users only, but they are useful if you must make drastic changes to your Terraform configuration. Drastic changes could be that you must split a configuration (and state files) into several pieces, or you are combining several configurations (and state files) into one.

Using terraform state is the recommended approach if you must edit your state file in some way. In the previous lesson I mentioned that we should never open and edit the state file directly, and here is the alternative that you should use instead.

Let's see what we can do with terraform state:

$ terraform state -h
Usage: terraform [global options] state <subcommand> [options] [args]

...

Subcommands:
    list                List resources in the state
    mv                  Move an item in the state
    pull                Pull current state and output to stdout
    push                Update remote state from a local state file
    replace-provider    Replace provider in the state
    rm                  Remove instances from the state
    show                Show a resource in the state

Many of these subcommands require that you specify the resource spec address that we saw in the previous section. For instance, continuing with the previous configuration with my imported storage account:

$ terraform state list

azurerm_storage_account.my_storage_account

The output shows I have a single resource and it gives me its resource spec address. I can view it in more detail:

$ terraform state show azurerm_storage_account.my_storage_account

# azurerm_storage_account.my_storage_account:
resource "azurerm_storage_account" "my_storage_account" {
    access_tier                       = "Hot"
    account_kind                      = "StorageV2"
    account_replication_type          = "LRS"
    account_tier                      = "Standard"
    ...
}

The output is identical to terraform show from before. Likewise I could run terraform state mv to move the resource from one resource spec address to another, for instance if I wanted to move the resource into a module or to a different local name like so:

$ terraform state mv \
    azurerm_storage_account.my_storage_account azurerm_storage_account.destination_account

Or I could delete a resource from my state if I no longer wish Terraform to manage it:

$ terraform state rm azurerm_storage_account.my_storage_account

In the certification exam I got a few questions regarding these commands, but no large difficult scenarios. If you know about resource addresses and you are familiar with terraform state mv, terraform state rm, terraform state list, and terraform state show you will be fine.

Use verbose logging for debugging purposes

Terraform is software, and like with any other software Terraform can have bugs or just behave in a way that you do not expect. If you need to know more details about what Terraform is doing during terraform init, terraform plan, terraform apply, and so on, you can control the log level using the TF_LOG environment variable.

Valid values for the TF_LOG environment variable is TRACE, DEBUG, INFO, WARN, or ERROR. TRACE has the highest verbosity, and ERROR has the lowest. A simple example of how to set this environment variable looks like this:

$ export TF_LOG=DEBUG
$ terraform init
...

If you want to control the log level for Terraform itself (Terraform core) and for all providers separately, you can do so using the two environment variables TF_LOG_CORE, and TF_LOG_PROVIDER.

If you think you have found a bug in Terraform you would usually be asked to provide detailed logs (TRACE or DEBUG) as part of your bug report.

Summary

This lesson was a mix of things that are considered more advanced usage of Terraform, but they are highly relevant in the real-world! In summary we looked at:

• How to import a resource into Terraform using terraform import <resource address> <resource id>, as well as how to view our current Terraform state using terraform show.
• How to manipulate our state in a safe way using terraform state and its subcommands:
  • terraform state list to list all resources in our state,
  • terraform state rm <resource address> to delete a resource from our state,
  • terraform state mv <source address> <destination address> to move a resource from one address to another address,
  • terraform state show <resource address> to show details about a resource in our state.
• How to control the output log level using the TF_LOG environment variable, setting it equal to TRACE, DEBUG, INFO, WARN, or ERROR.

1. ClickOps is where you use a mouse and you click in a graphical user interface to create the resources that you need. ↩

2. Note that this is a special case of the address to a resource, it could be more complicated if it is located in a module and if it is created using count or for_each. See the documentation at https://developer.hashicorp.com/terraform/cli/state/resource-addressing for all the details. ↩