⚠️Caution
Recently, managing EKS authentication via the API is preferred over using the aws-auth ConfigMap. Set authentication_mode to 'API' in the access configuration of the aws_eks_cluster resource, and streamline authentication management in Terraform by linking IAM roles with aws_eks_access_entry and aws_eks_access_policy_association.
REF
- https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/setting-up-access-entries.html
- https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_access_entry
- https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_access_policy_association
resource "aws_eks_cluster" "example" {
name = "example-cluster"
....
access_config {
authentication_mode = "API"
bootstrap_cluster_creator_admin_permissions = false
}
}
resource "aws_eks_access_entry" "example" {
cluster_name = aws_eks_cluster.example.name
principal_arn = aws_iam_role.example.arn
kubernetes_groups = ["group-1", "group-2"]
type = "STANDARD"
}
resource "aws_eks_access_policy_association" "example" {
cluster_name = aws_eks_cluster.example.name
policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
principal_arn = aws_iam_user.example.arn
access_scope {
type = "namespace"
namespaces = ["example-namespace"]
}
}
Introduction
Hi, everyone.
I would like to leave a memorandum about how to manage Kubernetes's configmap AWS auto-generated with terraform.
What Trouble
- If we want to add iam user/role for eks cluster operation, we need to fix auto-generated aws-auth configmap(namespace:kube-system)
- If we follow the official manual: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html), we should manage it with kubernetes manifest yml, but we want to manage it with terraform.
- Because at first we can access our eks cluster only with IAM user/role used when creating cluster(with ~/.kube/config as below) and our cluster generated role is terraform user/role
- Therefore, We want to add user/role to aws-auth configmap with terraform user/role and manage aws-auth configmap with terraform.
- Caution: According to EKS Best Practices Guides - Security, we should create the cluster with a dedicated IAM role
# ~/.kube/config
- name: arn:aws:eks:ap-northeast-1:9999999999:cluster/eks-example
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- --region
- ap-northeast-1
- eks
- get-token
- --cluster-name
- eks-example
command: aws
env: null
provideClusterInfo: false
How to resolve the trouble
1. Add terraform aws-auth configmap resouece and Use terraform import
command
1.0 Prepare terraform kubernetes provider
provider "kubernetes" {
host = data.aws_eks_cluster.eks.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.eks.token
}
- Caution: we should set host/token/cluster_ca_certificate to operate kubernetes cluster with tf-provider.
1.1 Prapare aws-auth configmap tf resource for importing
# aws-auth.tf
resource "kubernetes_config_map" "aws-auth" {
data = {
"mapRoles" = ""
}
metadata {
name = ""
namespace = ""
}
}
1.2 Execute terraoform import
cmd
terraform import kubernetes_config_map.aws-auth kube-system/aws-auth
1.3 terraform plan and remove diff from real resource state to resource config
resource "kubernetes_config_map" "aws-auth" {
data = {
"mapRoles" = <<EOT
- rolearn: arn:aws:iam::99999999999:role/hoge-role
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
# Therefore, before you specify rolearn, remove the path. For example, change arn:aws:iam::<123456789012>:role/<team>/<developers>/<eks-admin> to arn:aws:iam::<123456789012>:role/<eks-admin>. FYI:https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting_iam.html#security-iam-troubleshoot-ConfigMap
EOT
}
metadata {
name = "aws-auth"
namespace = "kube-system"
}
}
2. Fix aws-auth configmap resource we imported and add iam user/role
resource "kubernetes_config_map" "aws-auth" {
data = {
"mapRoles" = <<EOT
- rolearn: arn:aws:iam::99999999999:role/hoge-role
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
# Therefore, before you specify rolearn, remove the path. For example, change arn:aws:iam::<123456789012>:role/<team>/<developers>/<eks-admin> to arn:aws:iam::<123456789012>:role/<eks-admin>. FYI:https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting_iam.html#security-iam-troubleshoot-ConfigMap
# Add as below
- rolearn: hoge
username: hoge
groups: # REF: https://kubernetes.io/ja/docs/reference/access-authn-authz/rbac/
- hoge
EOT
}
metadata {
name = "aws-auth"
namespace = "kube-system"
}
}
Top comments (1)
Hi,
Thanks for the post.
Do you know a way to deploy k8s resources in a fully private eks cluster(control plane in private subnet) from terraform/ terraform cloud? So that I can edit the configmap from terraform itself