Vault Auto-unseal using Transit Secret Engine on Kubernetes

Theoretical introduction

To make the Vault operational once it has been installed, we need to perform two actions:

• Intialzie Vault
• Unseal Vault

Unsealing has to happen every time Vault starts. This is because Vault starts with sealed state in which it can't read storage because it doesn't know how to decrypt it.

Initialzing happens once when the server started with new backend. During initialization Vault generates bunch of keys:

• unseal keys
• encryption keys
• root token

As you can easily guess, unseal keys generated during initizlization are used for unsealing the Vault.

We can distinguish three options for unsealing the Vault:

• Manual unsealing
• Auto-unseal
• Transit Unseal - de facto one of Auto-unseal option

Manual unsealing

Manual unsealing is the simplest and doesn't require any additional configuration. Vault generated root key (don’t be confused with root token) and uses an algorithm  Shamir's Secret Sharing to split the key into chunks. During initialization, we can determine how many key shares will be needed to unseal the Vault. This is cloud agnostic and very flexible option but can become painful when you have many Vault clusters, many keys, and many key holders.

Auto-unseal

Auto-unseal reduces operational complexity and makes management less painful. In this approach, we delegate the responsibility of securing the unseal key from users to a trusted device or service. Vault with Auto-unseal, takes care of unsealing itself so we no longer need to worry about this as long as the service we have configured is available.

Services and devices supported with Auto-unseal you can find in official docs.

Transit Auto-unseal

We said that is one of Auto-unseal option. What makes this Auto-unseal different is that we don’t rely on an external service, but on external Vault itself. In this way, we can place one central Vault that will be responsible for Auto-unsealing other Vault instances.

Transit Auto-unseal setup

We are going to isolate our Vaults on the level of namespaces, so let's start with their creation.

# namespace for Vault central
kubectl create ns vault

# namespace for Vault with Transit Auto-unseal
kubectl create ns vault-a

Let’s start with installation of Vault Central. Save below Helm chart values to install Vault in HA mode with default manual unsealing.

server:
  affinity: ""
  ha:
    enabled: true
    replicas: 2
    raft: 
      enabled: true

# change namespace to Vault central
kns vault

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault -f vault-central-helm-values.yml

With the below initialization we split the root key into 4 shares (unseal keys). We can also set how many keys are required to reconstruct the root key, which is then used to decrypt the Vault's encryption key.

kubectl exec vault-0 -- vault operator init \
    -key-shares=4 \
    -key-threshold=2 \
    -format=json > vault-central-keys.json

Once is applied Vault generates unseal keys encrypted with base64 and hex and root token responsible for authentication against Vault. We will use it later.

{
  "unseal_keys_b64": [
    "4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T",
    "miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK",
    "EyVw9nQH/T+3zsa4HbPJ2s15l6B5MizMKQlKqs9taFzX",
    "zc7eU9MEvy9AaV4FPSQe7Jla2LcqSjS8KNPFDlQs0Rcg"
  ],
  "unseal_keys_hex": [
    "e169b9058b0d6a5fb331bb1bddec0d6e2eb32ed288397cf72fe3455fb8f0d3fdd3",
    "9a26ac837d4598f26ac7d2eb9cf69512e1badfd7ef8c0a99177829e1994ac3ec0a",
    "132570f67407fd3fb7cec6b81db3c9dacd7997a079322ccc29094aaacf6d685cd7",
    "cdcede53d304bf2f40695e053d241eec995ad8b72a4a34bc28d3c50e542cd11720"
  ],
  "unseal_shares": 4,
  "unseal_threshold": 2,
  "recovery_keys_b64": [],
  "recovery_keys_hex": [],
  "recovery_keys_shares": 0,
  "recovery_keys_threshold": 0,
  "root_token": "hvs.NbXRWfYNI4PmA860aBlC4onU"

Let’s unseal the first of two Vault instances in the central cluster. Pass two of four unseal_keys_b64 to the Vault to unseal them according to key-threshold.

kubectl exec vault-0 -- vault operator unseal 4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T
kubectl exec vault-0 -- vault operator unseal miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK

This same we have to do with the second instance, but before that, we must connect the second Vault to Raft storage cluster.

kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200

kubectl exec vault-1 -- vault operator unseal 4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T
kubectl exec vault-1 -- vault operator unseal miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK

Time to create Transit Secret Engine. This component will generate root key that we will use to Auto-unseal other Vaults.

# separate window
kubectl port-forward vault-0 -n vault 8200:8200

# set Vault address to use locally Vault CLI
export VAULT_ADDR=http://127.0.0.1:8200

# use 'root_token' generated during Vault initialization
vault login

# create transit secret 
vault secrets enable transit
vault write -f transit/keys/autounseal

Save below Vault policy that we will attach to Auto-unseal token.

path "transit/encrypt/autounseal" {
   capabilities = [ "update" ]
}

path "transit/decrypt/autounseal" {
   capabilities = [ "update" ]
}

# create policy with the above definition
vault policy write autounseal autounseal-policy.hcl

# create token for Auto-unsealing
$ vault token create -orphan -policy=autounseal -period=24h

Key                  Value
---                  -----
token                hvs.CAESIP_A7TaC9kt4yUeqg5_bJNiOJElb4UbA01xoV9Rk4ei6Gh4KHGh2cy5zVXpaa3A1MG9uOEZrNXN2a3J0TGl0cHU
token_accessor       wkTM4nsF0ehkRvIuBD9cedHC
token_duration       24h
token_renewable      true
token_policies       ["autounseal" "default"]
identity_policies    []
policies             ["autounseal" "default"]

Finally, we have created periodic orphan token which we will use for Auto-unsealing. Orphan means that created token doesn’t have a parent token so can’t be revoked together with ancestor. What is important to note, transit Auto-unseal token is renewed automatically by default.

Now it’s time to prepare Helm chart with the second Vault installation. It will be Vault with transit Auto-unsealing configuration. Check below Helm values file. Provide Vault central address and of course generated in previous step token (root key).

server:
  standalone:
    enabled: true
    config: |
      disable_mlock = true
      ui=true

      storage "file" {
        path = "/vault/data"
      }

      listener "tcp" {
        address     = "127.0.0.1:8200"
        tls_disable = "true"
      }

      seal "transit" {
        address = "http://vault.vault:8200"
        token = "hvs.CAESIP_A7TaC9kt4yUeqg5_bJNiOJElb4UbA01xoV9Rk4ei6Gh4KHGh2cy5zVXpaa3A1MG9uOEZrNXN2a3J0TGl0cHU"
        disable_renewal = "false"
        key_name = "autounseal"
        mount_path = "transit/"
        tls_skip_verify = "true"
      }

# change namespace to Vault Auto-unseal
kns vault-a

helm install vault hashicorp/vault -f vault-auto-unseal-helm-values.yml

The last step is to initialize Vault.

kubectl exec -it vault-0 -- vault operator init

Recovery Key 1: FFMLznSZq9wh/0CJwKLJWKkI9BrK/hjF6ySDYl9a19Ie
Recovery Key 2: qRfrdpkuEcXsF+dFh1Geru8VHkiL/hWUW+vY25twlwT1
Recovery Key 3: dX8sed7Dv8kI8kfFuYWDeQlagoikEVBpV5lZqH4ORnEh
Recovery Key 4: TCCplv+KvZHEOlICQU6eb67hGccufiqcZGkSiGlpQPkx
Recovery Key 5: ictL+c9czgMO+ME8qoTcGpgsvymEcORN7MkrpDE28x4a

Initial Root Token: hvs.6umGyyta9xrjq0q7Cv09Hr8X

Success! Vault is initialized

… and check its status to verify Sealed status.

kubectl exec -it vault-a -- vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.12.0
Build Date               2022-10-10T18:14:33Z
Storage Type             file
Cluster Name             vault-cluster-7a11a0ae
Cluster ID               a883d977-e70a-6367-3148-9c7a2c246897
HA Enabled               false

Each Vault initialization with configured to Auto-Unseal generates Recovery keys instead of Unseal Keys. Recovery keys can’t be used for unsealing Vault. These keys perform only authorization functions, which allows, for example, generates a new root token.

Final Thoughts

In the DevOps world, we want to automate everything possible and reduce operational complexity anywhere possible. Undoubtedly, Auto-unseal is something that fits into this assumption. However, from a security point of view, it is sometimes good to introduce a manual step with human intervention. In the above sample of Transit Auto-unseal we have exact combination of both approaches - Unsealed manually central cluster and related clusters Auto-unsealed by it.
It is also worth noting, that our solution is cloud-agnostic. We don’t rely on any external service, so we can setup it on-premise. The downside here is the introduction of a very crucial component in the overall deployment - central Vault cluster. Definitely, we have to think, how to ensure high availability and fault tolerance here.