DEV Community

Cover image for Trigger CI using Terraform Cloud
Kamesh Sampath
Kamesh Sampath

Posted on • Updated on

Trigger CI using Terraform Cloud

Continuous Integration(CI) pipelines needs a target infrastructure to which the CI artifacts are deployed. The deployments are handled by CI or we can leverage Continuous Deployment pipelines. Modern day architecture uses automation tools like terraform, ansible to provision the target infrastructure, this type of provisioning is called IaC.

Usually CI/CD and IaC don't run in tandem. Many times we want to trigger the CI pipeline only when the target infrastructure is ready to bootstrap with software components that are required by CI/CD pipelines.

As part of this DIY blog let us tackle the aforementioned problem with an use case.

Use Case

As CI/CD user I would like to provision a Kubernetes Cluster on Google Cloud Platform(GKE) using Terraform. The successful provision of the cluster should notify a CI pipeline to start bootstrapping ArgoCD on to GKE.

Architecture Overview

What you need ?

  • A Terraform Cloud Account. Create a workspace on the terraform cloud to be used for this exercise.
  • Google Cloud Account used to create the Google Kubernetes Engine(GKE) cluster.
  • Though we can use any CI platform, for this demo we will use Harness CIas our CI platform. You can do a free tier signup from here.

Demo Sources

The demo uses the following git repositories a sources,

  • IaC vanilla-gke: the terraform source repository that will be used with terraform cloud to provision GKE.
  • Kubernetes manifests bootstrap-argocd: the repository that holds kubernetes manifests to bootstrap argo CD on to the GKE cluster
  • Harness CI Pipeline tfc-notification-demo

Fork and Clone the Sources

To make fork and clone easier we will use gh CLI. Download the add gh to your $PATH.

Let us create a directory where we want to place all our demo sources,

mkdir -p "$HOME/tfc-notification-demo"
cd "$HOME/tfc-notification-demo"
export DEMO_HOME="$PWD"
Enter fullscreen mode Exit fullscreen mode

IaC

Clone and fork vanilla-gke repo,

gh repo clone harness-apps/vanilla-gke
cd vanilla-gke
gh repo fork
export TFC_GKE_REPO="$PWD"
Enter fullscreen mode Exit fullscreen mode

Bootstrap Argo CD Sources

Clone and fork bootstrap-argocd repo,

cd ..
gh repo clone harness-apps/bootstrap-argocd
cd bootstrap-argocd
gh repo fork
export ARGOCD_BOOTSTRAP_REPO="$PWD"
Enter fullscreen mode Exit fullscreen mode

Harness CI Pipeline

Clone and fork tfc-notification-demo repo,

cd ..
gh repo clone harness-apps/tfc-notification-demo
cd tfc-notification-demo
gh repo fork
export TFC_DEMO_REPO="$PWD"
Enter fullscreen mode Exit fullscreen mode

For rest of the blog we will reference the repositories vanilla-gke and bootstrap-argocd and tfc-notification-demo as $TFC_GKE_REPO, $ARGOCD_BOOTSTRAP_REPO and $TFC_DEMO_REPO.

Harness CI

In the following sections we will define and create the resources required to define a CI pipeline using Harness platform.

Create Harness Project

Create new harness project named terraform_integration_demos using Harness Web Console,

New Harness Project

Update its details as shown,

New Harness Project Details

Follow the wizard leaving rest to defaults and on the last screen choose Continuous Integration,

Use CI

Click Go to Module to go to project home page.

Define New Pipeline

Click Pipelines to define a new pipeline,

Get Started with CI

For this demo will be doing manual clone, hence disable the clone,

Disable

Click on Pipelines and delete the default Build pipeline,

Delete Pipeline

Add harnessImage Docker Registry Connector

As part of pipelines we will be pulling image from DockerHub. harnesImage Docker Registry Connector helps pulling the public Docker Hub images as an anonymous user.

Let us configure an harnesImage connector as described in docker registry connectors. The pipelines we create as part of the later section will use this connector.

Configure GitHub

GitHub Credentials

Create a GitHub PAT for the account where you have have forked the repositories $TFC_GKE_REPO and $ARGOCD_BOOTSTRAP_REPO. We will refer to the token as $GITHUB_PAT.

From the Project Setup click Secrets,

New Text Secret

Update the encrypted text secret details as shown,

GitHub PAT Secret

Click Save to save the secret,

Project Secrets

Connector

As we need to clone the sources from GitHub, we need to define a GitHub Connector, from the Project Setup click Connectors,

New Connector

From connector list select GitHub,

New GitHub Connector

Enter the name as GitHub,

GitHub Connector Overview

Click Continue to enter the connector details,

GitHub Connector Details

Click Continue and update the GitHub Connector credentials,

GitHub Connector Credentials

When selecting the Personal Access Token make sure you select the GitHub PAT that we defined in previous section,

GitHub PAT Secret

Click Continue and use select Connect through Harness Platform,

Connect through Harness Platform

Click Save and Continue to run the connection test, if all went well the connection should successful,

GH Connection Success

Google Cloud Service Account Secret

We need Google Service Account(GSA) credentials(JSON Key) to query the GKE cluster details and create resources on it.

Set environment

export GCP_PROJECT="the Google Cloud Project where Kubernetes Cluster is created"
export GSA_KEY_FILE="path where to store the key file"
Enter fullscreen mode Exit fullscreen mode

Create SA

gcloud iam service-accounts create gke-user \
  --description "GKE User" \
  --display-name "gke-user"
Enter fullscreen mode Exit fullscreen mode

IAM Binding

Add permissions to the user to be able to provision kubernetes resources,

gcloud projects add-iam-policy-binding $GCP_PROJECT \
  --member="serviceAccount:$GSA_NAME@$GCP_PROJECT.iam.gserviceaccount.com" \
  --role="roles/container.admin"
Enter fullscreen mode Exit fullscreen mode

Download And Save GSA Key

IMPORTANT: Only security admins can create the JSON keys. Ensure the Google Cloud user you are using has Security Admin role.

gcloud iam service-accounts keys create "${GSA_KEY_FILE}" \
    --iam-account="gke-user@${GCP_PROJECT}.iam.gserviceaccount.com"
Enter fullscreen mode Exit fullscreen mode

GSA Secret

Get back to the Project Setup click Secrets,

New File Secret

Add the GSA secret details as shown,

GSA Secret Details

IMPORTANT: When you browse and select make sure you select the $GSA_KEY_FILE as the file for the secret.

Click Save to save the secret,

Project Secrets

Terraform Workspace

On your terraform cloud account create a new workspace called vanilla-gke. Update the workspace settings to use Version Control and make it point to $TFC_GKE_REPO.

TFC Workspace VCS

Configure the workspace with following variables,

TFC Workspace Variables

For more details on available variables, check Terraform Inputs.

IMPORTANT: The GOOGLE_CREDENTIALS is Google Service Account JSON Key with permissions to create GKE cluster. Please check the https://github.com/harness-apps/vanilla-gke#pre-requisites for the required roles and permissions. This key will be used by Terraform to create the GKE cluster. When you add the key to terraform variables, you need to make it as base64 encoded e.g. cat YOUR_GOOGLE_CREDENTIALS_KEY_FILE | tr -d \\n

Going forward we will refer to the Terraform Workspace as $TF_WORKSPACE

Lookup your terraform cloud organizations

TFC Cloud Organization

And set it's value to the variable $TF_CLOUD_ORGANIZATION.

Create need Terraform API Token that can be used to pull the outputs of terraform run(cloud). From your terraform user settings Create an API token,

Terraform API Token

And save the API token to the variable $TF_TOKEN_app_terraform_io. We will use this variable in CI pipeline.

Harness CI Pipeline

Getting back to Harness web console, navigate to your project terraform_integration_demos, click Pipelines and Create a Pipeline --> Import From Git,

New CI Pipeline Import

Update the pipeline details as shown,

Pipeline Details

IMPORTANT: Make sure the Name of the pipeline is bootstrap argocd pipeline to make the import succeed with defaults.

Pipeline Import Successful

Click the bootstrap argocd pipeline from the list to open the Pipeline Studio and click on the stage Bootstrap Argo CD to bring up the pipeline steps,

Pipeline Steps

You can click on each step to see the details.

The Pipeline uses the following secrets,

  • google_application_credentials - the GSA credentials to manipulate GKE
  • terraform_cloud_api_token - the value of $TF_TOKEN_app_terraform_io
  • terraform_workspace - the value $TF_WORKSPACE
  • terraform_cloud_organization - the value $TF_CLOUD_ORGANIZATION

We already added google_application_credentials secret as part of the earlier section. Following the similar pattern let us add the terraform_cloud_api_token, terraform_workspace and terraform_cloud_organization as text secrets.

HINT:
From the Project Setup click Secrets,

New Text Secret

all terraform secrets

TIP: You can also skip adding terraform_workspace and terraform_cloud_organization, we can extract the values from the webhook payload using the expressions <+trigger.payload.workspace_name> and <+trigger.payload.organization_name> respectively.

Notification Trigger

For the Harness CI pipelines to listen to Terraform Cloud Events we need to define a Trigger, navigate back to pipelines and select the bootstrap argocd pipeline --> Triggers,

Pipeline Triggers

Click Add New Trigger to add a new webhook trigger(Type: Custom),

Custom Webhook Trigger

On the Configuration page enter the name of the trigger to be tfc notification,

TFC Notification Config

Leave rest of the fields to defaults and click Continue, leave the Conditions to defaults and click Continue.

On the Pipeline Input update the Pipeline Reference Branch to be set to main

Pipeline Input

NOTE: The Pipeline Reference Branch does not have any implication with this demo as we do manual clone of resources.

Click Create Trigger to create and save the trigger.

Trigger List

Copy Webhook URL

Webhook URL

Let us refer to this value as $TRIGGER_WEBHOOK_URL.

Terraform Notification

On your terraform cloud console navigate to the workspace Settings --> Notifications,

TFC Notifications

Click Create Notification and select Webhook as the Destination,

Webhook

Update the notification details as shown,

TFC Webhook Details

Since we need to bootstrap argo CD only on create events we set the triggers to happen only on Completed,

Trigger Events

Click Create Notification to finish the creation of notification.

TFC Webhook Creation Success

NOTE: The creation would have fired a notification, if the cluster is not ready yet the pipeline would have failed.

Congratulations!!. With this setup any new or updates thats done to the $TFC_GKE_REPO will trigger a plan and apply on Terraform Cloud. A Completed plan will trigger the bootstrap argocd pipline to run and apply the manifests from $BOOTSTRAP_ARGOCD_REPO on the GKE cluster.

An example of successful pipeline run

Pipeline Success

Summary

By using the terraform notifications feature we were able to make the CI pipelines listen to IaC events and run the CI pipelines as needed.

Notification Pattern

Top comments (0)