This post is aimed at those who already use Terraform and Google Cloud, but haven't yet used Terraform to manage whole projects or environments. But it may serve as inspiration for those who aren't yet using Infrastructure-as-code

Introduction to Terraform

Before we get onto what this blog post is actually about, let me talk a sec about Terraform:

For those of you still setting up your VMs and cloud resources by hand, chances are you've encountered situations where you don't remember how you set things up, or you're faced with having to do the same set of repetitive tasks.

Fortunately, there's a much better way, known as Infrastructure-as-code, where instead of doing things by hand using the web-base control panel or command-line interface, you write out what infrastructure you want using a declarative language, and have a program automatically go set them up for you. The benefit here is many:

1. Your configuration is in code, and easier to look through than flip through hundreds of different configuration screens on a web-based interface.
2. Because your configuration is in code, it can be checked into version control, diffed, and collaborated on easily.
3. The program applying your configuration will check what you've described in code, and what's set up in the cloud, and only make the changes it needs to to make the two match. So you don't have to remember if you already did something - you just make sure the code correctly describes what's needed, and apply it.

One such program (or really a whole ecosystem) to do this is Terraform, which can manage AWS, Google Cloud, Azure, and a whole lot more. As an example, the syntax looks something like this:

resource "google_compute_instance" "api" {
  name         = "api"
  machine_type = "e2-small"
  zone         = "us-central1-a"
  tags         = [google_compute_firewall.api.name]

  metadata = {
    ssh-keys = "terraform:${file(var.infra_pub)}"
  }

  boot_disk {
    initialize_params {
      image = "ubuntu-docker-certbot"
    }
  }

  network_interface {
    network = google_compute_network.default.name

    access_config {
      // Ephemeral IP
    }
  }
}


resource "google_dns_record_set" "api" {
  name = "api.${google_dns_managed_zone.platform.dns_name}"
  type = "A"
  ttl  = 300
  managed_zone = google_dns_managed_zone.platform.name
  rrdatas = [google_compute_instance.api.network_interface.0.access_config.0.nat_ip]
}

This little chunk of code would cause a small Google Compute instance called api to appear, with firewall set up, SSH key added for later (automated) deployments via CI/CD, and various other network configuration. Importantly, at the same time, it adds the VM's public IP address to the DNS records, so that it can be accessed at api.whateveryourdomainis.com. No need to do any of this manually.

This is just a small snippet of automation and convenience possible. Terraform is great, and if you're still managing your Public Cloud infrastructure by hand, give it a go.

However... this post isn't actually about how to use Terraform. Instead, it's addressing a specific complexity: setting up Google projects to be able to use Terraform takes multiple steps, how do we automate the automation?

I'm going to talk specifically about doing this on Google Cloud.

The Problem

You see, for Terraform to do its thing, you need to do several things to your Google Cloud project:

1. Create a service account and credentials so that it has the permissions to go do the things you need it to do
2. Optionally create a bucket for it to store its state file in. This is necessary for collaboration
3. Enable the various APIs that need to be enabled (you have to do this anyway when manually doing stuff, the web console will just redirect you to the page that lets you enable the APIs)

Unfortunately Terraform can't automate this because until you've done these things, it can't access your account or store its state. So you have to do some of this manually.

While it's not a lengthy process to do, when the organization gets a bit more complex, you will want to begin using several projects to keep resources isolated - development, staging, production, maybe others. Now you have to repeat this process multiple times, or worse, any time you want to change a setting across all of them at once, you have to repeat yourself.

Wouldn't it be great to also automate the installation of your automation?

The Solution

As it turns out, you can. Terraform is able to create whole Google Cloud Projects. So working from a parent or admin project, Terraform can in turn spawn all the other projects that you want, and for each of them create the necessary service accounts, buckets, and enable APIs for a project-specific Terraform workspace to take over and do the rest.

Doing it this way, you can standardise your automation. And there are much better security and permissions controls. You can even have a production environment that no human has the permission to access, if you'd like (presumably though, the log files go somewhere), with infrastructure changes handled solely by committing infrastructure-as-code files, and code changes handled by CI/CD.

You do however have to set up Terraform once, but I guess that's acceptable, this likely is the one Terraform installation that'll stick with you as your "master control" throughout the lifetime of the company. For me, I put all the critical tasks on here that only the owners and highly trusted individuals of the company should have access to, such as the configuration of the root DNS (which contains MX and SPF/DMARC/DKIM records for company email), and secrets storage (like Vault). Things that you don't want someone to accidentally (or deliberately) mess up and take the company offline.

The process looks like as follows:

1. Set up Organization

Before anything starts, we need to create Google Cloud Organization, which you can only do with a G Suite account or Cloud Identity account. There's some information for that here

Once you do this, you'll be able to access the Organization account under Google Cloud console, and the IAm that's specific to the organization. It looks something like this:

Note how the sidebar options "Service Accounts", "Labels" and "Identity-Aware Proxy" are greyed out. This is because this is looking at an Organization, rather than a GCP Project.

Once you have the organization, note the Organization ID (it's a long number).

2. Set up Billing Account

Projects created by Terraform need to be attached to a billing account, so we need one that it has access to. It would be useful to unify this under the same organization, so you can import an existing Billing Account into the organization via the "Identity & Organization", there's a button you can use to request projects and billing accounts from their owners to move them into centralized management.

Or you could create a new one. Once you have done this and have a working billing account, note the Billing Account ID (it's a long hex string with dashes).

3. Manually create your admin project

Unfortunately you still need one project to be created manually so that you can create the credentials and state storage for Terraform. This is your one-project-to-rule-them-all. Only key organization people should have the permissions to access this project.

Also note down this Project ID, you'll need it later

4. Create a service account for Terraform

In the IAM for the admin project, go to Service Accounts and "Create Service Account". This service account you will grant permissions to allow it to create projects in the organization.

Once created, go into its options and select "Add Key" and "Create a New Key", and opt for JSON type. Save this somewhere on your disk. This key is super important, as it grants access to all the permissions you give to this service account.

Add Permissions for this service account inside IAM. This service account only needs limited permissions inside this project, you can give it "Editor" permissions if you're lazy and accept the risks of over-broad permissions, otherwise grant only the permissions it needs to fulfil its purpose (if you're only using this project for admin, then it doesn't need many other permissions beyond access to the state bucket).

5. Create the Bucket for storing Terraform state

Terraform needs to store its state somewhere. It can store it locally, but if you lose your computer or these files, then you lose terraform state, which can be potentially awkward. Storing the state in a bucket is a little better, and also allows you to collaborate on terraform easier.

The first bucket needs to be created by hand in this admin project. Future buckets in the projects that are created by Terraform can be created automatically.

Since this bucket isn't used much, pick a Regional type, with Uniform access rules. If you're keeping your permissions trimmed down, at this point you can assign the Terraform service account to have access to this bucket. If instead you gave the service account "Editor" permissions, then it will have access to this bucket by default.

6. Add Terraform service account to the Organization IAM

In the Organization IAM you can now add the Terraform service account created earlier.

And grant it "Billing Account User" (as it needs to assign projects to the billing account) and "Project Creator" (as it of course needs to create projects). Once the service account creates a project, it automatically becomes an "Owner" in that project, so you don't need to assign any more cross-organization permissions than these. But remember: this is a security liability because it means this service account has the permissions to nuke those projects. If it made them, it can unmake them.

You probably also want to add yourself, or an admin group as a project viewer so that you at least by default get visibility on the projects created by Terraform (otherwise you'll find the interesting situation where Terraform creates the project, but nobody has access to it.

7. Enable some APIs

You'll need to enable some APIs that Terraform needs access to. If you don't enable these, you'll simply get an error message telling you to go enable them, with a link. To save you time, these are the ones you'll probably need to enable:

• Cloud Resource Manager
• Cloud Billing API
• IAM API

There may be a few others that I've forgotten, or if they're different on my environment. You'll see an error with a link to go enable them. These can also be enabled through CLI, but how to use that is beyond the scope of this post.

Hopefully that's the last manual editing of Google Cloud accounts you'll ever need, the rest can be handled through automation

8. Setting up your environment

Make sure you have Terraform installed, and create a new repository where you'll store your IaC. Make sure you grab that credentials.json from an earlier step and put that somewhere on your system that's secure and only you can access.

9. Writing your IaC code

vars.tf

I like to keep even my terraform installation modular, so I have a single file that contains all the variables that make this my project. In theory if I start up another company, I can re-use the terraform code but change the variables in this vars.tf

variable "project" {
  description = "The name of the admin project"
  type = string
  default = "<name of your admin project>"
}

variable "region" {
  description = "Region where most things are being created"
  type = string
  default = "us-central1"
}

variable "zone" {
  description = "Zone where resources are created"
  type = string
  default = "us-central1-a"
}

variable "org_id" {
  description = "Organization ID under which all projects are created"
  type = string
  default = "<your Organization ID>"
}

variable "billing_id" {
  description = "Billing account ID to attach projects to"
  type = string
  default = "<your Billing account ID>"
}

Replace the values here with the various IDs and names created in previous steps.

main.tf

This is where I usually keep all the provider and backend definitions. The file contains this:

terraform {
  backend "gcs" {
    credentials = "<path to your credentials.json>"
    bucket  = "<name-of-the-bucket-you-created>"
    prefix  = "terraform/tfstate"
  }
}

provider "google" {
  credentials = "<path to your credentials.json>"
  project     = var.project
  region      = var.region
  zone        = var.zone
}

Replace the values here with the credentials ID location, and the bucket name you created

With just these two files, you're ready to initialize and try out Terraform. you can run terraform init and then terraform plan or even terraform apply (there should be nothing to apply at this point).

If you'd like, at this point you can do things like create VMs, set up DNS entries, etc. etc. for your admin account. But what we're really after is creating a whole new Google Project, complete with its own Terraform credentials ready for another Terraform workspace to take it further. That's the last step

10. Create a Google Project creation module

I'm going to put this in its own Terraform module, so that I can re-use the same setup for all my projects. These will live inside a folder called modules that sits one level above the Terraform workspace

../modules/project/vars.tf

Here are the variables that this module will need. Note, I don't supply the default values, as I expect them to be supplied when the module is loaded later

variable "project_name" {
  description = "Pick a Human-readable name of the project"
  type = string
}

variable "project_id" {
  description = "Pick a Project ID"
  type = string
}

variable "billing_id" {
  description = "Billing ID to attach project to"
  type = string
}

variable "org_id" {
  description = "Organization ID to attach project to"
  type = string
}

variable "region" {
  description = "Pick a region most things go into"
  type = string
}

variable "bucket_name" {
  description = "Pick a name for the terraform state storage bucket"
  type = string
}

../modules/project/main.tf

Here's the business end of this whole post

resource "google_project" "project" {
  name = var.project_name
  project_id = var.project_id
  billing_account = var.billing_id
  org_id = var.org_id
}

resource "google_project_service" "project" {
  project = google_project.project.project_id

  for_each = toset([
    "compute.googleapis.com",
    "cloudresourcemanager.googleapis.com",
    "iam.googleapis.com",
    "dns.googleapis.com",
  ])

  service = each.key

  disable_on_destroy = false
}

resource "google_service_account" "project_terraform" {
  # service account for project-specific terraform
  project = google_project.project.project_id
  account_id = "terraform"
  display_name = "${var.project_name} Terraform"
  description = "Terraform service account for ${var.project_name}"
}

resource "google_storage_bucket" "project_terraform" {
  # terraform state storage bucket
  project = google_project.project.project_id
  name = var.bucket_name
  location = var.region

  uniform_bucket_level_access = true
}

resource "google_storage_bucket_iam_member" "project_terraform" {
  bucket = google_storage_bucket.project_terraform.name
  role = "roles/storage.admin"
  member = "serviceAccount:${google_service_account.project_terraform.email}"
}

This sets up that Terraform bucket, enables APIs, and creates service accounts, exactly like we had to earlier manually. And thus we have automated the creation and setup of resources needed for Terraform to manage a project going forward.

Note: this is not a complete module. You will still need to create a service account key and store that somewhere, as well as grant this new service account the relevant permissions inside that project. I'm not going to cover that here, but you might want to start with:

resource "google_service_account_key" "project_terraform" {
  service_account_id = google_service_account.project_terraform.name
}

Caution: This will create a service account key, and store it in this terraform project's state storage. This may or may not be desired. It's certainly convenient - you can then simply inspect this value by hand using terraform show, or with additional stages, output this state using an output entry to display it on screen, local_file to save the file somewhere locally, or other providers to send that credential to some secrets storage. There are some security considerations to this that go beyond the scope of this post. If you're seriously creating this for your business, please read up about the security implications of storing secrets in the terraform state.

project_dev.tf

With the two module files defined, we can now use this module to create a dev environment:

module "dev" {
  source = "../modules/project"

  billing_id = var.billing_id
  org_id = var.org_id
  region = var.region

  project_name = "Development"
  project_id = "<pick a project ID>"
  bucket_name = "<pick a name for the bucket>"
}

project_prod.tf

Why stop at one project? Let's make another one out of the same module!

module "prod" {
  source = "../modules/project"

  billing_id = var.billing_id
  org_id = var.org_id
  region = var.region

  project_name = "Production"
  project_id = "<pick a project ID>"
  bucket_name = "<pick a name for the bucket>"
}

Once these files are up, you can simply terraform init (needed as new modules were defined) and terraform apply. Barring any errors and needing to go enable any APIs, and the one remaining issue of creating and downloading that service account credentials.json I touched upon above, Terraform should quickly set up your new projects with all the bits necessary for further automation!

Next steps

Your next steps are: Create your Terraform workspace for these environments. For example, you could maintain another repository of IaC (a different one to your Admin one which doesn't need to be touched again unless new projects are needed, or settings need to be changed), and within this repository, have a separate dev and prod branch. You can set up CI (or just run it manually) such that running terraform apply in the dev branch will target the dev project, while running terraform apply in the prod branch will target the prod project.

Doing it this way lets you run experimental development infrastructure using IaC from the dev branch, and when you're ready to go to production with this infrastructure, merge the IaC changes from dev branch into prod branch, and apply it from there.

Other things you might want to do are:

• Create a shared VPC in your admin project that allow resources from the other projects to communicate with each other
• Set up your root DNS in your admin project
• Set up your MX DKIM/DMARC/SPF records in the admin project
• Set up DNS and networks so that you have prod.yourdomain.com and dev.yourdomain.com
• Set up your other projects that handle backups, or company website, or research/sandbox environments, or staging environments

Cover Photo by Ant Rozetsky on Unsplash