Introduction
In a recent DSAG (German-speaking SAP User Group) event ("DSAG Betriebstage" - can be translated as Ops Days) I had the opportunity to give a talk about Infrastructure as Code (IaC) and how it can help you with governance topics on the SAP Business Technology Platform (BTP).
One important topic in this context was of course the automated detection of configuration drifts. The scenario basically was that an SAP BTP subaccount was (perfectly) set up using Terraform following all company-specific guidelines. However, a user with the necessary permissions changed the configuration manually in the SAP BTP cockpit. This led to a deviation between the automated configuration and the actual state of the subaccount. This is what we call a configuration drift. How can we find out about this drift?
While I showed how this can be detected via Terraform, we did not go into the technical details of the process. To make this theoretical description more tangible we will take a closer look on how we can achieve this from a technical perspective.
Prerequisites
To make the following examples work I leveraged the following tools and services:
- A GitHub repository for storing the Terraform configuration
- An Azure Blob Storage for centrally storing the Terraform state
- GitHub Actions to automate the Terraform workflow especially the drift detection
You can of course also use different backends like AWS S3. You find a complete list of supported backends in the Terraform documentation. The same is true for the CI/CD pipeline. You can use e.g., GitLab CI/CD or any other CI/CD tool you prefer to enable the automated flow.
The GitHub Repository
I created a GitHub repository that you find at https://github.com/btp-automation-scenarios/btp-terraform-drift. This repository contains the Terraform configuration for an SAP BTP subaccount.
The configuration is very basic and only creates a subaccount with a few entitlements to show the drift detection flow. The infrastructure configuration is stored in the infra
directory of the repository.
The sensitive information for the setup like username and password is stored in GitHub Secrets.
Setup for Remote State
The detection of a configuration drift relies on the storage of the Terraform state. This state file contains the configuration of the infrastructure and is used by Terraform to determine what must be changed in the infrastructure.
In a real-life setup this state is stored centrally to make it available to the team of BTP administrators. This can be an Azure Blob Storage, AWS S3 or any other supported backend. One important point here is that the state needs to be encrypted as it might contains sensitive information. In this blog post we will use an Azure Blob Storage.
The first step is to create such a storage on Microsoft Azure. The Microsoft documentation provides a very good guide on how to setup such a storage account for Terraform state. You find the documentation here.
For the sake of simplicity, we will shortly walk through the steps using the Azure CLI:
- Create a resource group on Azure:
az group create --name rg_terraform_state_sapbtp --location westeurope
- Create a storage account in the resource group:
az storage account create --resource-group rg_terraform_state_sapbtp --name sasapbtptfstate --sku Standard_LRS --encryption-services blob
- Create a blob container in the storage account:
az storage container create --name tfstate --account-name sasapbtptfstate
This results in the following setup for the storage account:
and the blob container (that already contains the state file):
You can also do the setup via the Azure Portal or Terraform.
Be aware that:
- Azure storage comes with automatic encryption. You can also use customer managed keys for that.
- Azure state storage blobs are automatically locked before any operation that writes to the storage account. This prevents concurrent writes to the state file. Details are described in the documentation.
Next, we must make Terraform aware of the fact that should store the state in the Azure Blob Storage. This is done by the following configuration to the provider.tf
file:
terraform {
required_providers {
btp = {
source = "sap/btp"
version = "~>1.1.0"
}
}
backend "azurerm" {
resource_group_name = "rg_terraform_state_sapbtp"
storage_account_name = "sasapbtptfstate"
container_name = "tfstate"
key = "dev.terraform.tfstate"
}
}
The relevant block is the backend
block. We use a predefined backend provided by Terraform for Azure. The resource_group_name
, storage_account_name
and container_name
are the values we used to define the Azure Blob Storage. The key
is the name of the state file.
Next, we define the workflows to create and destroy the subaccount.
GitHub Actions for the Setup
To execute the creation and the deletion via GitHub Actions I added two workflows to the repository:
-
.github/workflows/setup-subaccount.yml
: this workflow executed the setup of the subaccount in the SAP BTP based on some input variables. The configuration is given by:
name: Basis Subaccount via Terraform
on:
workflow_dispatch:
inputs:
PROJECT_NAME:
description: "Name of the project"
required: true
default: "sample-proj-drift"
REGION:
description: "Region for the sub account"
required: true
default: "eu10"
COST_CENTER:
description: "Cost center for the project"
required: true
default: "1234567890"
STAGE:
description: "Stage for the project"
required: true
default: "DEV"
ORGANIZATION:
description: "Organization for the project"
required: true
default: "B2B"
env:
PATH_TO_TFSCRIPT: 'infra'
jobs:
execute_base_setuup:
name: BTP Subaccount Setup
runs-on: ubuntu-latest
steps:
- name: Check out Git repository
id: checkout_repo
uses: actions/checkout@v4
- name: Setup Terraform
id : setup_terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
terraform_version: latest
- name: Terraform Init
id: terraform_init
shell: bash
run: |
export export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} init -no-color
- name: Terraform Apply
id: terraform_apply
shell: bash
run: |
export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
export BTP_USERNAME=${{ secrets.BTP_USERNAME }}
export BTP_PASSWORD=${{ secrets.BTP_PASSWORD }}
terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} apply -var globalaccount=${{ secrets.GLOBALACCOUNT }} -var region=${{ github.event.inputs.REGION }} -var project_name=${{ github.event.inputs.PROJECT_NAME }} -var stage=${{ github.event.inputs.STAGE }} -var costcenter=${{ github.event.inputs.COST_CENTER }} -var org_name=${{ github.event.inputs.ORGANIZATION }} -auto-approve -no-color
-
.github/workflows/destroy-subaccount.yml
: this workflow destroys the subaccount in the SAP BTP. The configuration is given by:
name: Destroy Subaccount via Terraform
on:
workflow_dispatch:
env:
PATH_TO_TFSCRIPT: 'infra'
jobs:
execute_base_setuup:
name: BTP Subaccount Deletion
runs-on: ubuntu-latest
steps:
- name: Check out Git repository
id: checkout_repo
uses: actions/checkout@v4
- name: Setup Terraform
id : setup_terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
terraform_version: latest
- name: Terraform Init
id: terraform_init
shell: bash
run: |
export export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} init -no-color
- name: Terraform Apply
id: terraform_apply
shell: bash
run: |
export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
export BTP_USERNAME=${{ secrets.BTP_USERNAME }}
export BTP_PASSWORD=${{ secrets.BTP_PASSWORD }}
terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} destroy -var globalaccount=${{ secrets.GLOBALACCOUNT }} -auto-approve -no-color
There is one important thing to note here. The Terraform CLI needs to authenticate not only to SAP BTP but also to Azure to access the Blob storage. I used the storage access key for that. This key is stored in the GitHub Secrets along with the other sensitive information. To make the access key available to Terraform we must export it in the Terraform steps as ARM_ACCESS_KEY
.
With these GitHub Actions we can setup and destroy the subaccount in the SAP BTP including a central storage of the Terraform state in the Azure Blob Storage. The next step is to automate the detection of a configuration drift.
Drift Detection
The detection of a configuration drift is done by comparing the actual state of the subaccount in the SAP BTP with the state defined in the Terraform configuration. This sounds complicated, but indeed it is quite easy.
The only thing we need to do is to execute a terraform plan
inside of a GitHub Action. If the stored state matches the configuration on SAP BTP the planning will basically state "nothing to do". In case there is a deviation the planning will come up with changes that it would like to apply to bring the state and the configuration on SAP BTP back in sync. We use this to find out if any unwanted changes have been made.
To make things easier for the handling in a CI/CD flow we provide an additional parameter to the terraform plan
command namely the -detailed-exitcode
. This parameter tells the CLI to provide more granular information about what the resulting plan via the exit code:
- 0 = Succeeded with empty diff, so no drift detected
- 1 = Error - this should not happen, but at least good to know that things went completely of the rails
- 2 = Succeeded with non-empty diff, so changes are present, and a drift is detected.
With this we create a new workflow for the drift detection:
name: Check for Subaccount Drift via Terraform
on:
workflow_dispatch:
inputs:
PROJECT_NAME:
description: "Name of the project"
required: true
default: "sample-proj-drift"
REGION:
description: "Region for the sub account"
required: true
default: "eu10"
COST_CENTER:
description: "Cost center for the project"
required: true
default: "1234567890"
STAGE:
description: "Stage for the project"
required: true
default: "DEV"
ORGANIZATION:
description: "Organization for the project"
required: true
default: "B2B"
env:
PATH_TO_TFSCRIPT: 'infra'
jobs:
execute_base_setuup:
name: BTP Subaccount Drift Check
runs-on: ubuntu-latest
steps:
- name: Check out Git repository
id: checkout_repo
uses: actions/checkout@v4
- name: Setup Terraform
id : setup_terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
terraform_version: latest
- name: Terraform Init
id: terraform_init
shell: bash
run: |
export export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} init -no-color
- name: Terraform Plan
id: terraform_plan
shell: bash
continue-on-error: true
run: |
export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
export BTP_USERNAME=${{ secrets.BTP_USERNAME }}
export BTP_PASSWORD=${{ secrets.BTP_PASSWORD }}
terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} plan -var globalaccount=${{ secrets.GLOBALACCOUNT }} -var region=${{ github.event.inputs.REGION }} -var project_name=${{ github.event.inputs.PROJECT_NAME }} -var stage=${{ github.event.inputs.STAGE }} -var costcenter=${{ github.event.inputs.COST_CENTER }} -var org_name=${{ github.event.inputs.ORGANIZATION }} -no-color -detailed-exitcode
- name: Create issue
if: steps.terraform_plan.outcome == 'failure'
uses: actions/github-script@v7
env:
PROJECT_NAME: ${{ github.event.inputs.PROJECT_NAME }}
REGION: ${{ github.event.inputs.REGION }}
STAGE: ${{ github.event.inputs.STAGE }}
COST_CENTER: ${{ github.event.inputs.COST_CENTER }}
RUN_ID : ${{ github.run_id }}
with:
script: |
const issueTitle = `Configuration Drift Detected for ${process.env.PROJECT_NAME}`
const issueBody = `A drift has been detected for ${process.env.PROJECT_NAME} in ${process.env.REGION} region. Stage is ${process.env.STAGE} and cost center is ${process.env.COST_CENTER}. Find more information in the run https://github.com/btp-automation-scenarios/btp-terraform-drift/actions/runs/${process.env.RUN_ID}`
github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
labels: [
'automated issue', 'drift detected'
],
title: issueTitle,
body: issueBody
})
- name: State deviation - Set run to failed
if: steps.terraform_plan.outcome == 'failure'
uses: actions/github-script@v7
with:
script: |
core.setFailed('A configuration drift was detected!')
The main part of the workflow is the step with the id terraform plan
that executes the planning and provides us the exit code that we use to check if a drift is detected. As follow-up actions we:
- Create an issue in the GitHub repository to inform the team about the drift
- Set the run to failed to also make it visible in the GitHub Actions overview
The step for the creation is defined as:
- name: Create issue
if: steps.terraform_plan.outcome == 'failure'
uses: actions/github-script@v7
env:
PROJECT_NAME: ${{ github.event.inputs.PROJECT_NAME }}
REGION: ${{ github.event.inputs.REGION }}
STAGE: ${{ github.event.inputs.STAGE }}
COST_CENTER: ${{ github.event.inputs.COST_CENTER }}
RUN_ID : ${{ github.run_id }}
with:
script: |
const issueTitle = `Configuration Drift Detected for ${process.env.PROJECT_NAME}`
const issueBody = `A drift has been detected for ${process.env.PROJECT_NAME} in ${process.env.REGION} region. Stage is ${process.env.STAGE} and cost center is ${process.env.COST_CENTER}. Find more information in the run https://github.com/btp-automation-scenarios/btp-terraform-drift/actions/runs/${process.env.RUN_ID}`
github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
labels: [
'automated issue', 'drift detected'
],
title: issueTitle,
body: issueBody
})
This step is executed only if the plan state failed which we ensure via the if condition if: steps.terraform_plan.outcome == 'failure'
.
We use the GitHub REST API to create an issue with the corresponding labels. As an example, we provide some additional information, but you can of course also provide more detailed information about the detected drift.
Make sure that within the settings of the repository (or the organization) the GitHub Actions are allowed to create issues by enabling the "Read and write permissions" for GitHub Actions:
In the current state the workflow would be reported as "successful". I personally prefer that the run is marked as failed in case of a drift. This must be done explicitly by the following step:
- name: State deviation - Set run to failed
if: steps.terraform_plan.outcome == 'failure'
uses: actions/github-script@v7
with:
script: |
core.setFailed('A configuration drift was detected!')
With this we have an automated drift detection in place that informs the team about any deviations between the actual state of the subaccount in the SAP BTP and the configuration defined in the Terraform configuration.
Once we introduce a drift, we will see the following in the GitHub Actions overview:
As we can see the exit code is as expected a "2" indicating that changes would be applied.
And consequently, an issue is created in the GitHub repository:
Conclusion and Outlook
With only minor configurations the Terraform built-in functionalities of state storage and planning enable us to detect configuration drifts in the SAP BTP to automatically detect deviations of the configuration with respect to the company-specific guidelines. This ensures that configuration is not changed without the knowledge of the administrator team especially to ensure governance and compliance.
Now that we have detected the drift, how to deal with it? This usually cannot be handled automatically but needs to be analyzed in detail. You have basically two options that bring things back in sync:
- Reconcile the state with the configuration in the Terraform configuration. This can be done by executing a
terraform apply
. This will apply the necessary changes to the infrastructure and bring it back in sync with the state. - The manual changes on the platform were okay (e.g., due to an emergency fix). In this case the state file must be updated to reflect the actual state. This can be done by executing a
terraform apply
and aterraform apply
with the-refresh-only
flag. This will update the state file with the actual state of the subaccount and not do any changes to the infrastructure.
Another aspect worth to discuss is the integration into existing processes especially on the SAP side of the house. We implemented the flow and the "handling" (namely the creation of a GitHub issue) in an isolated fashion. However, we also received the ask to integrate the flow into the SAP solutions that might already deal with such governance tasks like SAP Cloud ALM. based on that we have a feature request open (link) in the repository of the Terraform Provider for SAP BTP. If you would like to share your point of view or vote for it, this is the place to go.
With this, nothing more to say than: Happy Terraforming!
Top comments (0)