AWS: Kubernetes - AWS Secrets Manager and Parameter Store integration
Storing access data in Kubernetes Secrets has an important drawback, because they are only available within the Kubernetes cluster itself.
To make them available to external services, we can use Hashicorp Vault and integrate it with Kubernetes using solutions such as vault-k8s
or use services from AWS - Secrets Manager or Parameter Store.
Integrating AWS Secrets Manager and Parameter Store into Kubernetes will give us the ability to create a new type of resource — SecretProviderClass
which we can connect to Kubernetes Pods as files or environment variables.
For this, we will need AWS Secrets and Configuration Provider (ASCP) and Kubernetes Secrets Store CSI Driver.
AWS Secrets and Configuration Provider vs Hashicorp Vault
I haven’t used Vault for a long time, but regarding the question “What to use”, it’s a choice between the “setup, configure, and manage Hashicorp Vault yourself” (installation of the Helm chart and configuration of accesses) or “use a ready-made solution from AWS” (in fact, you only need to configure IAM roles).
Also, keep in mind that using AWS services (surprise!) is paid, so if you plan to have thousands of secrets, it’s probably better to use Vault.
In addition, Vault itself provides much more opportunities, for example, the generation of temporary tokens for services, plus, as far as I remember — Kubernetes Pods can receive parameters from Vault without the need to create Kubernetes Secrets, while when using AWS Secrets and Configuration Provider and Kubernetes Secrets Store, CSI Driver for connecting variables will create Kubernetes Secrets.
However, our project already uses Secrets Manager and Parameter Store, so I don’t see a point in Vault (yet), so let’s integrate our existing secrets into an AWS Elastic Kubernetes Service cluster.
AWS Secrets Manager vs Parameter Store
You can read more about the difference between them here — AWS — Difference between Secrets Manager and Parameter Store (Systems Manager), here just briefly.
Common features:
- both use AWS KMS to encrypt data
- both are Key/Value Store
- both support versioning
Key differences:
- Cost :
- Secrets Manager: Charges $0.40 for each secret and $0.05 for every 10,000 API requests
- Parameter Store: for the Standard, it does not take money for storage, with higher throughput it costs $0.05 for every 10,000 API requests, with Advanced parameters — $0.05 for storage and $0.05 for every 10,000 API requests
- Secrets rotation :
- Secrets Manager: has a built-in rotation mechanism and integrates it with services (RDS, DocumentDB, etc)
- Parameter Store: You must implement the rotation yourself
- Cross-account Access :
- Secrets Manager: Supports
- Parameter Store: Not supported
- Cross-Regions Replication :
- Secrets Manager: Supports
- Parameter Store: Not supported
- Data limits :
- Secrets Manager: up to 10KB per secret
- Parameter Store: 4KB for each record (8KB for Advanced Parameters)
- Quantity limits:
- Secrets Manager: 500,000 per region and account
- Parameter Store: 10,000 per region and account
Installing Secrets Store CSI Driver
So, for integration, we need two services — Secrets Store CSI Driver and AWS Secrets and Configuration Provider.
First, we’ll add the Secrets Store CSI Driver.
With its help, we will be able to connect secrets/parameters from AWS as files or variables to Kubernetes Pods.
Add a Helm chart and install it with the syncSecret.enabled=true
parameter to create Kubernetes Secrets and synchronize them with AWS secrets during data rotation (see Sync as Kubernetes Secret ):
$ helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
$ helm -n kube-system install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver — set syncSecret.enabled=true
Check Pods:
$ kubectl -n kube-system get pod | grep secret
csi-secrets-store-secrets-store-csi-driver-kzmcx 3/3 Running 0 31s
csi-secrets-store-secrets-store-csi-driver-t7bqc 3/3 Running 0 31s
Installing AWS Secrets and Configuration Provider
Add a repository and install the Helm chart:
$ helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
$ helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws
Check Pods:
$ kubectl -n kube-system get pod | grep secret
csi-secrets-store-secrets-store-csi-driver-kzmcx 3/3 Running 0 9m
csi-secrets-store-secrets-store-csi-driver-t7bqc 3/3 Running 0 9m
secrets-provider-aws-secrets-store-csi-driver-provider-awskq5g8 1/1 Running 0 23s
secrets-provider-aws-secrets-store-csi-driver-provider-awsksq9d 1/1 Running 0 23s
And look at the CSIDriver:
$ kubectl get csidriver
NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH MODES AGE
ebs.csi.aws.com true false false <unset> false Persistent 46h
efs.csi.aws.com false false false <unset> false Persistent 4d
secrets-store.csi.k8s.io false true false <unset> false Ephemeral 10m
Next, we will configure IAM for IRSA.
IAM Policy та IAM Role для ServiceAccount
In order for Kubernetes Pods to be able to access AWS SecretManager and Parameter Store, we will use IRSA — create a ServiceAacount that will use an IAM Role with an IAM Policy that will have permissions to call Secrets Manager and Parameter Store (see AWS: EKS, OpenID Connect, and ServiceAccounts).
Describe an IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"ssm:DescribeParameters",
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:GetParametersByPath"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
Create it in the IAM:
$ aws iam create-policy --policy-name ascp-iam-policy --policy-document file://ascp-policy.json
{
“Policy”: {
“PolicyName”: “ascp-policy”,
“PolicyId”: “ANPAXFIUAIGSBPFEDKZZT”,
“Arn”: “arn:aws:iam::492***148:policy/ascp-iam-policy”,
…
Describe the Trust policy for the IAM Role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRoleWithWebIdentity",
"Principal": {
"Federated": "arn:aws:iam::492 ***148:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/2DC*** 124"
},
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/2DC***124:aud": "sts.amazonaws.com",
"oidc.eks.us-east-1.amazonaws.com/id/2DC***124:sub": "system:serviceaccount:default:ascp-test-serviceaccount"
}
}
}
]
}
Create the Role itself with this trust policy:
$ aws iam create-role --role-name ascp-iam-role --assume-role-policy-document file://ascp-trust.json
{
“Role”: {
“Path”: “/”,
“RoleName”: “ascp-iam-role”,
“RoleId”: “AROAXFIUAIGSLDCB3L4AR”,
“Arn”: “arn:aws:iam::492***148:role/ascp-iam-role”,
…
Attach the ascp-iam-policy
policy to that role:
$ aws iam attach-role-policy --role-name ascp-iam-role --policy-arn=arn:aws:iam::492***148:policy/ascp-iam-policy
Now we can create a SecretProviderClass and a Pod that will use it.
Create a SecretProviderClass
Let’s add a SecretProviderClass, which will receive a string from the Secrets Manager and a string from the Parameter Store, and then we will connect them to a Kubernetes Pod.
Create a secret in Secrets Manager:
$ aws secretsmanager create-secret --name ascp-secret-test-string --secret-string "secretLine"
{
“ARN”: “arn:aws:secretsmanager:us-east-1:492***148:secret:ascp-secret-test-string-DNweNg”,
“Name”: “ascp-secret-test-string”,
“VersionId”: “9d4f490d-edcc-4ee0-b43d-5b4e25fa271b”
}
Create a record in the Parameter Store:
$ aws ssm put-parameter --name ascp-ssm-test-param --value "paramLine" --type "String"
{
“Version”: 1,
“Tier”: “Standard”
}
Next, describe a SecretProviderClass with two objects - in the parameters.objects.objectName
set a name of the object in the Secrets Manager or Parameter Store and in the objectType
set where we get this object from:
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: aspc-test-secret-class
spec:
provider: aws
parameters:
objects: |
- objectName: "ascp-test-string"
objectType: "secretsmanager"
- objectName: "ascp-ssm-test-param"
objectType: "ssmparameter"
Let’s go to the Pod.
Attaching a SecretProviderClass to a Pod as a file
Add a ServiceAccount with the IAM role we created earlier, and a Pod with this ServiceAccount:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ascp-test-serviceaccount
namespace:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::492***148:role/ascp-iam-role
---
apiVersion: v1
kind: Pod
metadata:
name: ascp-test-pod
spec:
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
command: ['sleep', '36000']
volumeMounts:
- name: ascp-test-secret-volume
mountPath: /mnt/ascp-secret
readOnly: true
restartPolicy: Never
serviceAccountName: ascp-test-serviceaccount
volumes:
- name: ascp-test-secret-volume
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: aspc-test-secret-class
Deploy it:
$ kubectl apply -f ascp-test.yaml
serviceaccount/ascp-test-serviceaccount created
secretproviderclass.secrets-store.csi.x-k8s.io/aspc-test-secret-class created
pod/ascp-test-pod created
Check the Pod:
$ kk describe pod ascp-test-pod
…
Mounts:
/mnt/ascp-secret from ascp-test-secret-volume (ro)
…
Volumes:
…
ascp-test-secret-volume:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: secrets-store.csi.k8s.io
FSType:
ReadOnly: true
VolumeAttributes: secretProviderClass=aspc-test-secret-class
…
And a content of the /mnt/ascp-secret
directory:
$ kk exec -ti ascp-test-pod -- ls -l /mnt/ascp-secret
total 8
-rw-r — r — 1 root root 10 Jul 17 09:32 ascp-secret-test-string
-rw-r — r — 1 root root 9 Jul 17 09:32 ascp-ssm-test-param
And files there:
$ kk exec -ti ascp-test-pod -- cat /mnt/ascp-secret/ascp-secret-test-string
secretLine
$ kk exec -ti ascp-test-pod -- cat /mnt/ascp-secret/ascp-ssm-test-param
paramLine
Attaching a SecretProviderClass to a Pod as an environment variable
Mounting secrets as a file may be a good solution for some .env
files, but what about environment variables? For example to pass a DB_PASSWORD
.
To do so, we can add a secretObjects
to the SecretProviderClass - then Kubernetes Secrets Store CSI Driver will create a Kubernetes Secret, which we can connect to the Pod:
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: aspc-test-secret-class
spec:
provider: aws
parameters:
objects: |
- objectName: "ascp-secret-test-string"
objectType: "secretsmanager"
- objectName: "ascp-ssm-test-param"
objectType: "ssmparameter"
secretObjects:
- secretName: aspc-test-kube-secret
type: Opaque
data:
- objectName: ascp-secret-test-string
key: kube-secret-key
Here:
-
secretObjects.secretName
: a name of the Kubernetes Secret to be created -
secretObjects.secretName.data.objectName
: must be the same as parameters.objects.objectName -
secretObjects.secretName.data.key
: a key for the Kubernetes Secret -data.kube-secret-key
Deploy, and check Kubernetes Secret:
$ kk get secret aspc-test-kube-secret -o yaml
apiVersion: v1
data:
kube-secret-key: c2VjcmV0TGluZQ==
…
And a value of the kube-secret-key
:
$ echo c2VjcmV0TGluZQ== | base64 -d
secretLine
Now, let’s connect it to the Pod — add spec.containers.env
with the valueFrom.secretKeyRef
:
---
apiVersion: v1
kind: Pod
metadata:
name: ascp-test-pod
spec:
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
command: ['sleep', '36000']
env:
- name: SECRET
valueFrom:
secretKeyRef:
name: aspc-test-kube-secret
key: kube-secret-key
volumeMounts:
- name: ascp-test-secret-volume
mountPath: /mnt/ascp-secret
readOnly: true
restartPolicy: Never
serviceAccountName: ascp-test-serviceaccount
volumes:
- name: ascp-test-secret-volume
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: aspc-test-secret-class
Deploy, and check:
$ kk exec -ti ascp-test-pod -- printenv | grep SECRET
SECRET=secretLine
Or:
$ kk exec -ti ascp-test-pod -- bash
bash-4.2# echo $SECRET
secretLine
During this, we have to attach the volumes
and volumeMounts
, as it was done for mounting Secrets as a file.
Creating a SecretProviderClass from JSON
If a data in Secrets Manager and Parameter Store is stored in JSON, then we should use the jmesPath
in our SecretProviderClass.
Let’s create another secret in Secrets Manager from JSON:
$ aws secretsmanager create-secret --name ascp-secret-test-json --secret-string '{"username":"admin", "password":"foobar"}'
{
“ARN”: “arn:aws:secretsmanager:us-east-1:492***148:secret:ascp-secret-test-json-iOtcBf”,
“Name”: “ascp-secret-test-json”,
“VersionId”: “32666608–6416–46cf-8b93-cf090eef1bc5”
}
Check it:
Update our SecretProviderClass:
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: aspc-test-secret-class
spec:
provider: aws
parameters:
objects: |
- objectName: "ascp-secret-test-string"
objectType: "secretsmanager"
- objectName: "ascp-ssm-test-param"
objectType: "ssmparameter"
- objectName: "ascp-secret-test-json"
objectType: "secretsmanager"
jmesPath:
- path: "username"
objectAlias: "ascp-test-username"
- path: "password"
objectAlias: "ascp-test-password"
secretObjects:
- secretName: aspc-test-kube-secret
type: Opaque
data:
- objectName: ascp-secret-test-string
key: kube-secret-key
- secretName: aspc-test-kube-secret-json
type: Opaque
data:
- objectName: ascp-test-username
key: kube-secret-user
- objectName: ascp-test-password
key: kube-secret-pass
Here:
- in the
parameters.objects.objectName: "ascp-secret-test-json"
usejmesPath
, which parses our secret and receives the values of two fields -username
andpassword
, for which it creates twoobjectAlias
- ascp-test-username and ascp-test-password - in the
secretObjects.secretName: aspc-test-kube-secret-json
we add adata
with twoobjectName
, in which we useobjectAlias
from the parameters
Update our Kubernetes Pod — add two secretKeyRef
with the kube-secret-user
and kube-secret-user
keys from the aspc-test-kube-secret-json
Secret:
---
apiVersion: v1
kind: Pod
metadata:
name: ascp-test-pod
spec:
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
command: ['sleep', '36000']
env:
- name: SECRET
valueFrom:
secretKeyRef:
name: aspc-test-kube-secret
key: kube-secret-key
- name: USER
valueFrom:
secretKeyRef:
name: aspc-test-kube-secret-json
key: kube-secret-user
- name: PASS
valueFrom:
secretKeyRef:
name: aspc-test-kube-secret-json
key: kube-secret-pass
volumeMounts:
- name: ascp-test-secret-volume
mountPath: /mnt/ascp-secret
readOnly: true
restartPolicy: Never
serviceAccountName: ascp-test-serviceaccount
volumes:
- name: ascp-test-secret-volume
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: aspc-test-secret-class
Deploy and check Kubernetes Secrets:
$ kk get secret
NAME TYPE DATA AGE
aspc-test-kube-secret Opaque 1 2s
aspc-test-kube-secret-json Opaque 2 2s
And a value from the aspc-test-kube-secret-json
:
$ kk get secret aspc-test-kube-secret-json -o yaml
apiVersion: v1
data:
kube-secret-pass: Zm9vYmFy
kube-secret-user: YWRtaW4=
…
And environment variables in the Pod:
$ kk exec -ti ascp-test-pod --printenv | grep 'SECRET\|USER\|PASS'
SECRET=secretLine
USER=admin
PASS=foobar
Done.
Originally published at RTFM: Linux, DevOps, and system administration.
Top comments (0)