Terraform & HashiCorp Vault Integration: Seamless Secrets Management

what is Hashicorp vault.?

Imagine HashiCorp Vault as a secure digital vault for all your sensitive information like passwords, API keys, and encryption certificates. It acts like a central location to store, access, and manage these secrets. Here's why it's useful:

• Strong Security: Vault encrypts your secrets and controls access with different permission levels. This minimizes the risk of unauthorized access and keeps your sensitive information safe.

• Centralized Management: No more scattered secrets! Vault keeps everything in one place, making it easier to control and audit who can access what.

• It can store various secrets, from database credentials to cloud API keys. Vault also integrates with different tools and platforms you might already be using.

Overall, using Vault helps organizations improve security, simplify secret management, and streamline access control for critical information. While Vault has a free open-source version with limited features, most businesses opt for the paid edition with additional functionalities like advanced auditing and disaster recovery

-1-. In today's demo, we will explore Terraform Vault integration with a real example and troubleshooting. Let's get started!

-2-. Launch one EC2 instance with basic configuration and access it.

-3-. Then update your machine and install GPG as well.

sudo apt update && sudo apt install gpg

-4-. Then download the signing key to a new keyring.

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

-5-. Then verify the key's fingerprint.

gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint

-6-. Clone the Vault repository from GitHub.

mkdir -p $GOPATH/src/github.com/hashicorp && cd $_
git clone https://github.com/hashicorp/vault.git
cd vault

-7-. Then install Vault.

snap install vault

-8-. To start Vault, you can use the following command:

vault server -dev -dev-listen-address="0.0.0.0:8200"

-9-. You may need to set the following environment variables. To do so, create a duplicate session and copy the command, then paste it into the new session.

export VAULT_ADDR='http://0.0.0.0:8200'

-10- Then go to your EC2 instance's security group and open port 8200 to access the Vault UI.

-11-. And then copy the public IP and paste it into your browser's address bar followed by port 8200. You will see the login page appear.

-12-. For logging in as root, select 'Token' as the default method. Copy the token from your server when you start Vault; it will be displayed to you. After that, you will see the dashboard.

-13-. Then, in this demo, we will store the secret as 'Key & value'. To do so, click on 'Secret Engines', then select 'KV' which stands for 'Key/value', and click on 'Next'.

-14-. Next, on the following window, provide a path and click 'Next' to proceed, where you will notice that no credentials are created or stored.

-15-. Now, click on 'Create Secret' and input your secret. I will use 'Username & password' as an example.

-16-. Then, to access these credentials in Terraform, you need to assign policies or roles similar to IAM. Click on 'Policies', then click on 'Enable New Method', and select 'App Role' as the authentication method. This is similar to IAM. Enable the role. Using this App Role, I will authenticate Terraform or whichever tool you are using.

-17-. Then, create a duplicate session of your machine and run the following command on it.

-18-. Because HashiCorp Vault does not support the creation of users in the UI, go to the console and create a user. It's very easy. Before creating the role, we need to create a policy for that role. This policy enables Terraform to access the 'kv' and 'secret' folders in Vault.

vault policy write terraform - <<EOF
path "*" {
  capabilities = ["list", "read"]
}

path "secrets/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "kv/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "secret/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "auth/token/create" {
capabilities = ["create", "read", "update", "list"]
}
EOF

-19-. Now, create a role.

vault write auth/approle/role/terraform \
    secret_id_ttl=10m \
    token_num_uses=10 \
    token_ttl=20m \
    token_max_ttl=30m \
    secret_id_num_uses=40 \
    token_policies=terraform

-20-. Similar to AWS, in Vault, we have Role ID and Secret ID. This is sensitive information, so do not share it with anyone.

vault read auth/approle/role/terraform/role-id

vault write -f auth/approle/role/terraform/secret-id

-21-. Now that we are done with Vault, the next step is to write down the Terraform project and check whether Terraform is able to read the secret from Vault.

#main.tf

provider "aws" {
  region = "us-east-1"
}

provider "vault" {
  address = "http://<your-ip_address>:8200"
  skip_child_token = true

  auth_login {
    path = "auth/approle/login"

    parameters = {
      role_id = "<your-role_id>"
      secret_id = "<your-secret_id>"
    }
  }
}

data "vault_kv_secret_v2" "example" {
  mount = "kv" // change it according to your mount
  name  = "test-secret" // change it according to your secret
}

-22-. Then, to check if Terraform is able to retrieve the credentials from Vault, run the 'terraform init' and 'terraform apply' commands.

-23-. Now, let's add the block to create an EC2 instance and apply it again. You will see that in the EC2 tag, our password will be taken.

#main.tf

provider "aws" {
  region = "us-east-1"
}

provider "vault" {
  address = "http://<your-ip_address>:8200"
  skip_child_token = true

  auth_login {
    path = "auth/approle/login"

    parameters = {
      role_id = "<your-role_id>"
      secret_id = "<your-secret_id>"
    }
  }
}

data "vault_kv_secret_v2" "example" {
  mount = "kv" // change it according to your mount
  name  = "test-secret" // change it according to your secret
}

resource "aws_instance" "example" {
  ami = "ami-0c7217cdde317cfec"
  instance_type = "t2.micro"

  tags = {
    secret = data.vault_kv_secret_v2.example.data["username"]
  }
}

-24-. If you encounter the following error while running 'terraform apply' command, then run it again to recreate 'role-id' and 'secret-id' with the same command as mentioned in step no. 20. After that, paste the new credentials into the code. This will solve your problem.