DEV Community

Cover image for Scheduled backup of Vault secrets with Jenkins on Kubernetes
Igor Zhivilo
Igor Zhivilo

Posted on

Scheduled backup of Vault secrets with Jenkins on Kubernetes

#vault #jenkins #kubernetes #hashicorp

I am a DevOps engineer at Cloudify.co and I will share in this post my experience related to automation of Vault backup creation using Jenkins scheduled job and simple python script which I built to create dump of vault secrets.

Let's start.

What is HashiCorp's Vault?

Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more. Vault provides a unified interface to any secret while providing tight access control and recording a detailed audit log.
https://www.vaultproject.io/

Prerequisites:

  • Vault Installed
  • Jenkins Installed

My Setup

  • EKS Kubernetes cluster
  • Vault runs on EKS cluster
  • Jenkins runs on EKS cluster

You can read in this tutorial how to run Jenkins on EKS cluster:
https://igorzhivilo.com/jenkins/ci-cd-future-k8s-jenkins

What you will learn from this post?

  • How to use python hvac library for authentication with Vault programmatically and backup vault secrets.

  • What is AppRole authentication mechanism in Vault and how to enable/create it.

  • How to create scheduled backup for Vault secrets with Jenkins pipeline on k8s.

AppRole authentication method in Vault

How can an application programmatically request a token so that it can read secrets from Vault?

Using the AppRole which is an authentication mechanism within Vault to allow machines or apps to acquire a token to interact with Vault and using the policies you can set access limitations for your app.

It uses RoleID and SecretID for login.

role-secret

Create AppRole and policy for Jenkins

I explained how to do it in detail in my blog post: https://igorzhivilo.com/jenkins/how-to-read-vault-secrets-from-declarative-pipeline

After you applied everything I wrote in this post:

  • enabled approle in vault 
  • v2 kv secrets engine enabled
  • applied all needed policies

eventually you will get role_id and secret_id which will be used programmatically with 'python hvac'.

Another way for 3rd step (apply all needed policies) is to create policy using Vault's UI:

$ kubectl port-forward -n vault svc/vault 8200
Forwarding from 127.0.0.1:8200 -> 8200
Forwarding from [::1]:8200 -> 8200
Enter fullscreen mode Exit fullscreen mode

Go to policy tab -> Create ACL Policy

acl_policy

path "sys/auth/approle" {
  capabilities = [ "create", "read", "update", "delete", "sudo" ]
}
path "sys/auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}
path "auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "secret/data/jenkins/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "secret/metadata/jenkins/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
Enter fullscreen mode Exit fullscreen mode

and then run via vault CLI:

$ vault write auth/approle/role/jenkins token_policies=jenkins \
 token_ttl=1h token_max_ttl=4h

# Get RoleID and SecretID
$ vault read auth/approle/role/jenkins/role-id
$ vault write -f auth/approle/role/jenkins/secret-id
Enter fullscreen mode Exit fullscreen mode

Test that you created correctly role_id/secret_id 

$ vault write auth/approle/login \
    role_id=ROLE_ID \
    secret_id=SECRET_ID
Enter fullscreen mode Exit fullscreen mode

Testing authentication with vault using python hvac and appRole

Simple python script to test auth with vault:

import hvac

VAULT_URL = 'http://vault.vault.svc.cluster.local:8200'
client = hvac.Client(url=VAULT_URL)
client.auth.approle.login(
    role_id = self.role_id,
    secret_id = self.secret_id
)

assert client.is_authenticated()
Enter fullscreen mode Exit fullscreen mode

first run 'pip install hvac'.

I am running this script from pod with python container inside of my Kubernetes cluster. 

URL of vault in k8s cluster: 'http://vault.vault.svc.cluster.local:8200'

You will see authentication error if authentication is failed, if you do, make sure you applied all the needed policies, enabled applrole, and generated properly role_id / secret_id.

Validate role_id/secret_id is correct using vault CLI:

$ vault write auth/approle/login \
    role_id=YOU_ROLE_ID \
    secret_id=YOU_SECRET_ID
Enter fullscreen mode Exit fullscreen mode

Get the list of secrets under 'jenkins' vault_prefix (CLI)

In my case, vault_prefix looks like: 'secret/data/jenkins' and all secrets stored under 'jenkins' prefix:

$ vault kv list secret/jenkins

Keys
----
aws
git
web-app1
web-app2
...
Enter fullscreen mode Exit fullscreen mode

Each key in list has additional subset of keys, for example 'aws' has access_key_id/secret_access_keys

Getting the secrets list (python) 

secrets_list_response = client.secrets.kv.v2.list_secrets(path = 'jenkins')

print('The following keys are available under "jenkins" prefix: {keys}'.
  format(keys=','.join(secrets_list_response['data']['keys'])))
Enter fullscreen mode Exit fullscreen mode

If you have a permissions error on the secrets list, check you have access to metadata, that what you should see in UI for 'jenkins policy':

path "secret/metadata/jenkins/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
Enter fullscreen mode Exit fullscreen mode

If you don't, add using the UI or vault CLI:

$ tee jenkins-policy-metadata.hcl <<"EOF"
  path "secret/metadata/jenkins/*" {
    capabilities = [ "read" ]
  }
  EOF

$ vault policy write jenkins jenkins-policy-metadata.hcl
Enter fullscreen mode Exit fullscreen mode

Get a specific secret (python) 

secret_response = client.secrets.kv.v2.read_secret(path = 'jenkins/aws')

print(secret_response)
Enter fullscreen mode Exit fullscreen mode

If you have a permission error, check you have access to data in UI of Vault:

path "secret/data/jenkins/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
Enter fullscreen mode Exit fullscreen mode

VaultHandler

I created VaultHandler which you can find here.

https://github.com/warolv/vault-backup

You can use it to:

  • Dump all your secrets as encrypted yaml/json files, or you can store it without encryption.
  • Get a list of your secrets.
  • Print all secrets nicely.
  • Populate Vault from yaml/json dumps to a specific 'vault_prefix'.

Also, I think to extend it to use different auth methods, besides appRole, create CLI, to run it in the command line and much more :-)

If the idea sounds interesting, add stars to the repo or clone it, I will know this way you like the idea.

Create Jenkins scheduled job for daily vault backup 

def configuration = [vaultUrl: "${VAULT_URL}",  vaultCredentialId: "vault-role-app", engineVersion: 2]

def secrets = [
  [path: 'secret/jenkins/aws', engineVersion: 2, secretValues: [
    [envVar: 'AWS_ACCESS_KEY_ID', vaultKey: 'aws_access_key_id'],
    [envVar: 'AWS_SECRET_ACCESS_KEY', vaultKey: 'aws_secret_access_key']]],
  [path: 'secret/jenkins/vault-backup', engineVersion: 2, secretValues: [
    [envVar: 'VAULT_ADDR', vaultKey: 'vault_url'],
    [envVar: 'ROLE_ID', vaultKey: 'role_id'],
    [envVar: 'SECRET_ID', vaultKey: 'secret_id'],
    [envVar: 'VAULT_PREFIX', vaultKey: 'vault_prefix'],
    [envVar: 'DUMP_ENCRYPTION_PASSWORD', vaultKey: 'encryption_password']]],
]

def podTemplate = """
                apiVersion: v1
                kind: Pod
                spec:
                  containers:
                    - name: awscli
                      image: amazon/aws-cli
                      command:
                      - cat
                      tty: true
                    - name: python
                      image: python:3.6
                      command:
                      - cat
                      tty: true
                """.stripIndent().trim()

pipeline {
  agent {
    kubernetes {
      defaultContainer 'jnlp'
      yaml "${podTemplate}"
    }
  }

  environment {
    AWS_DEFAULT_REGION = "eu-west-1"
  }

  stages {
    stage('Backup Jenkins'){
      steps {
        container('python'){
          dir("${env.WORKSPACE}/pipelines-k8s/vault-backup/") {
            withVault([configuration: configuration, vaultSecrets: secrets]){
              sh """#!/bin/bash
                pip install -r requirements.txt
                python -u vault_handler.py
                tar -zcvf vault_secrets.json.enc.tar.gz vault_secrets.json.enc
              """
            }
          }
        }
        container('awscli'){
          dir("${env.WORKSPACE}/pipelines-k8s/vault-backup/") {
            withVault([configuration: configuration, vaultSecrets: secrets]){
              sh '''
                aws s3 cp vault_secrets.json.enc.tar.gz s3://vault-backups/$(date +%Y%m%d%H%M)/vault_secrets.json.enc.tar.gz
              '''
            }
          }
        }
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Now create a new pipeline in jenkins: newitem -> pipeline and make it periodic (daily).

new_pipeline

In this post, I described how to automate Vault backup creation using Jenkins scheduled job and simple python script which I built to create dump of vault secrets.

Thank you for reading, I hope you enjoyed it, see you in the next post.

If you want to be notified when the next post of this tutorial is published, please follow me on Twitter @warolv.

Original story on my blog: https://igorzhivilo.com/vault/scheduled-backup-vault-secrets/

Top comments (0)

Read next

osnibjunior profile image

KCD Events in 2025

Osni B. Junior -

adnanlatif profile image

From 41 Minutes to 8 Minutes: How I Made Our CI/CD Pipeline 5x Faster

Adnan Latif -

ptuladhar3 profile image

11th Dec 2024 — OpenAI Outage (ChatGPT) Explained: Kubernetes Clusters on Fire!

Puru -

suin profile image

Deploying Helm Charts to Multiple Kubernetes Clusters with Cluster API's HelmChartProxy

suin -