Shifting Left - Secret as Code with Flux V2 and Mozilla SOPS

Introduction

A question that every engineers faced during their software development lifecycle is, "where should i stored my secrets?"

In this article, I will be sharing how my team "shift-left" to allow engineers self-service in managing their secrets. The guiding principle we adhering to is being as cloud-agnostic as possible.

Concepts

Before we dive deeper, let's align some concepts first. When we discuss about secret management for application, there are various way secret can be managed.

1. Writing them directly on source code as plain text. The anti-pattern and bad security practice, although it will work.

2. Putting the secrets somewhere else, like Cloud Provider solutions, Vault, File Systems in the Platform or Env Variables stored in the CI/CD.

3. Keeping them in the same code repository in an encrypted manner. Now we are talking about secrets lifecycle being managed via a pull request in a git-based workflow. This is the topic of secret as code we are talking about, where our Git Repository is the source of truth for secrets and configs.

Getting started

Before we start, we need to bootstrap flux to an existing cluster first.

Follow the instruction here if you need to: https://fluxcd.io/docs/get-started/

For my experiment, these are the technologies i use

1. Flux v2
2. Gitlab
3. Microsoft Azure Kubernetes Service
4. Microsoft Key Vault
5. Azure CLI
6. SOPs

Context

Below are the steps on how we can bootstrap an existing Kubernetes cluster to have the capability of decrypting SOPS secrets.

For my experimentation, i will be using Azure Key Vault for cryptographic operations. It is also supported by other Cloud Providers Key Management Service, or self generated PGP/GPG Key.

High-Level Setup

Here are the summarised steps to give Flux the capability in performing cryptographic operations. Flux's controller will be able to decrypt SOPs secret whenever the secret is being consumed by a pod.

• Install Pod-Identity
• Create Role Assignments for Kubelet
• Create a managed identity
• Create Azure KeyVault and Signing Key
• Configure in-cluster secrets decryption

Detailed set-up :
https://techcommunity.microsoft.com/t5/azure-global/gitops-and-secret-management-with-aks-flux-cd-sops-and-azure-key/ba-p/2280068

Sample Repository:
https://github.com/robincher/bluesky-flux-sops-azure-template

Checking in Secrets

Before continuing , we need to install sops locally in our machine first.

• Create a sample secret

# sample-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: demo-credentials
  namespace: default
type: Opaque
stringData:
  username: admin
  password: xxxxx

• Encrypting the secret

1. Running the SOPs command with required parameters

sops -e --azure-kv <kv_key_address> --in-place --encrypted-regex "^(data|stringData)$" sample-secret.yaml > sample-secret-enc.yaml

Important flags to provide

a. azure-kv : Azure Keyvault Key Address, make sure you or the machine have the access

b. encrypyed-regex: Specifying what to be encrypted from the secret file.

1. Creating a default SOPS setting

Alternatively, we can create a default SOPS config that specify what to use for encryption, and where to be encrypted.

Create a .sops.yaml with the following values. We will encrypt all blocks under the stringData or data key.

# .sops.yaml
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    azure_keyvault: https://demo123.vault.azure.net/keys/demo-secret-key/c12231211

The default sops configs will be applied to its root and sub-folders.

Now we just have to run the following command to encrypt.

sops -e --in-place sample-secret.yaml > sample-secret-enc.yaml

You should get the following sample secret upon encryption

# sample-secret-enc.yaml
apiVersion: v1
kind: Secret
metadata:
    name: demoapp-credentials
    namespace: default
type: Opaque
stringData:
    username: ENC[AES256_GCM,data:21q4bo0=,iv:LOLxXQurjQR6cu9heQlZDdmhNgYO6VCBybbQHV6rO0w=,tag:58ep32CDrlCFuuDnD65VEQ==,type:str]
    password: ENC[AES256_GCM,data:oTZDkadQKL45dA==,iv:5VVbXC55xTVwH/n3t5gtKNtlkB3q7t8lW7Jw1czNSL0=,tag:WuqdubjTu6mQN5x1b3zDyw==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv:
        - vault_url: https://demo123.vault.azure.net
          name: demo-secret-key
          version: b7bc85c1a4ef4180be9d1de46725304c
          created_at: "2021-04-23T14:22:15Z"
          enc: KuFxRbcge198GU7hwHs078JNd_1EFtvcFqQ6bOLJDYMnWaW0kSbeD4DCxY0jX9MA17Rv3UMKHGfImgEbNfXGGIh7UucLPygpiuUyn9I73ClSQQ4trc4bD2yVkonCMwz5-0MiPVC3muhQpn3KjhThSucOgjhBnqQy_zzzzTeUP9PWi1pSp1jc3S2BxQIuKy09-oEakQogU4BRy55219befizYN7EFe8mstSIkvpksqGxKccH6dQum2k-OqsBUH2jkxiVgi5CEU35COy0pNWVJpZGuOaDMkGGqo7lrT4XKEGxtFKvEDxr6bTfjjQafuuxW9-4a9ZtaBkHCKopk66R9dcQ
    hc_vault: []
    age: []
    lastmodified: "2021-01-23T14:22:18Z"
    mac: ENC[AES256_GCM,data:as5mfREh5xdeiwbchkiiBS96tGuLJnEqme6VdDrPWKV9R0A4ATIM/1+HcbdAzGBXb8TmhO71hZMl3IvmX9DrNA/tvpPwFvLCkDfNhoWXJoXRRv6aRR7AJPlfcXkVMxxYaRDqz+ugAJkZG+5dhYeh1QAmiswjZOXaINEOw3Jf5dI=,iv:p/M2OhPdh2Naxu37Jt7EwiLf9Eb9OgExsmXX3hSUOJQ=,tag:fVqJ2jy++6GxHBPGXZHmHw==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.7.1

Testing

Let's check in the secret and see the magic by GitOps

rm sample-secret.yaml
git add sample-secret-enc.yaml
git commit -m "chore: Add encrypted secret"
git push

Check out the secret in Kubernetes

kubectl describe secret -n demoapp demoapp-credentials

Observation and Learning

• We need to further educate the developers this new way of doing as the majority of them are used to checking in secrets into an external secret manager or stored as CI/CD variables.

• Mozilla SOPs with Flux allow us to versioned our secrets as it follows a git-based workflow, and provide audit trails through the commit messages.

• It can be deployed universally in both cloud or on-premises environment, as long as the cryptographic keys are backup and kept safely. We acknowledge that using Azure Key Vault might required additional work for us if we are to move to another provider's Key Management Service.

• Using a cloud-based KMS alleviate the risk of having the decryption key being leaked (which typically being stored in the same machine), as there are access control in placed.

• This is not a silver bullet to address all kind of secrets requirements as what work well for us might be going against the way of working for yours.

References

1. Flux with SOPS
2. GitOps with AKS
3. Sample Repository Flux and AKS