DEV Community

John Ajera
John Ajera

Posted on

Why Your EBS CSI Driver Can’t Attach Volumes (And How IRSA Fixes It)

#eks #ebs #csi #irsa

If you're running EKS and trying to use the AWS EBS CSI Driver, you've probably hit a cryptic volume attachment error at some point. Here's a breakdown of the exact problem I faced, and how I resolved it using IAM Roles for Service Accounts (IRSA).

📊 The Problem

After installing the EBS CSI Driver via Helm, pods like Prometheus and others failed to bind their PersistentVolumeClaims (PVCs). On checking the controller logs:

kubectl logs -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver -c ebs-plugin --tail=100
Enter fullscreen mode Exit fullscreen mode

You may see an error like this:

Could not attach volume "vol-..." to node "i-...": operation error EC2: AttachVolume, https response error StatusCode: 403, api error UnauthorizedOperation
Enter fullscreen mode Exit fullscreen mode

🔍 How to Check Configuration

Use these commands to validate that everything is wired correctly:

✅ Controller logs (should NOT show AttachVolume 403 errors) 

kubectl logs -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver -c ebs-plugin --tail=100
Enter fullscreen mode Exit fullscreen mode

Bad output:

... AttachVolume ... UnauthorizedOperation ...
Enter fullscreen mode Exit fullscreen mode

Good output:

I0330 00:04:47.133164       1 node.go:944] "CSINode Allocatable value is set" nodeName="i-035ea5232976e526c" count=26
Enter fullscreen mode Exit fullscreen mode

✅ Validate CSINode registration 

kubectl get csinode
Enter fullscreen mode Exit fullscreen mode

Good output:

NAME                         DRIVER              ...
i-035ea5232976e526c          ebs.csi.aws.com     ...
Enter fullscreen mode Exit fullscreen mode

✅ Confirm node tolerations 

kubectl get daemonset ebs-csi-node -n kube-system -o jsonpath='{.spec.template.spec.tolerations}' | jq
Enter fullscreen mode Exit fullscreen mode

Expected:

[
  {"operator": "Exists"},
  {"effect": "NoExecute", "key": "node.kubernetes.io/not-ready", "operator": "Exists"},
  {"effect": "NoSchedule", "key": "node.kubernetes.io/disk-pressure", "operator": "Exists"}
]
Enter fullscreen mode Exit fullscreen mode

✅ Check IRSA annotations 

kubectl describe sa ebs-csi-controller-sa -n kube-system
Enter fullscreen mode Exit fullscreen mode

Expected:

Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/cltest-ebs-csi-controller-irsa
Enter fullscreen mode Exit fullscreen mode

✅ Inspect IAM Role and policies 

aws iam get-role --role-name cltest-ebs-csi-controller-irsa
aws iam list-attached-role-policies --role-name cltest-ebs-csi-controller-irsa
Enter fullscreen mode Exit fullscreen mode

Expected:

"PolicyName": "AmazonEBSCSIDriverPolicy"
Enter fullscreen mode Exit fullscreen mode

✅ Evaluate IAM policies

You can also simulate the action to confirm permissions:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::<ACCOUNT_ID>:role/cltest-ebs-csi-controller-irsa \
  --action-names ec2:AttachVolume ec2:DetachVolume \
  --resource-arns arn:aws:ec2:<REGION>:<ACCOUNT_ID>:instance/* \
  --region <REGION>
Enter fullscreen mode Exit fullscreen mode

Expected:

"EvalDecision": "allowed"
Enter fullscreen mode Exit fullscreen mode

🔐 IAM Inline Policy

Make sure this inline policy is also present:

resource "aws_iam_role_policy" "ebs_volume_permissions" {
  name = "allow-ebs-volume-ops"
  role = aws_iam_role.ebs_csi_controller_irsa.name

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Sid    = "EBSVolumeActions",
        Effect = "Allow",
        Action = [
          "ec2:AttachVolume",
          "ec2:DetachVolume"
        ],
        Resource = [
          "arn:aws:ec2:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:volume/*",
          "arn:aws:ec2:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:instance/*"
        ]
      }
    ]
  })
}
Enter fullscreen mode Exit fullscreen mode

💡 Use aws iam get-role-policy --role-name <name> --policy-name allow-ebs-volume-ops to verify.

🎉 Outcome

Once the proper inline policy was added, everything worked. Volumes were attached and Prometheus stopped showing PVC binding issues.

👋 Final Thoughts

The EBS CSI Driver depends on your IAM plumbing being just right. While AmazonEBSCSIDriverPolicy covers most use cases, sometimes additional permissions are needed.

Check your trust policies, policy attachments, and don't forget: inline IAM policies can help patch IAM gaps when needed.

Happy debugging!

🧪 Quick Summary of Checks

Check Command Expected Result
Controller logs kubectl logs -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver -c ebs-plugin --tail=100 No UnauthorizedOperation errors
CSINode registration kubectl get csinode Node lists ebs.csi.aws.com driver
Tolerations `kubectl get daemonset ebs-csi-node -n kube-system -o jsonpath='{.spec.template.spec.tolerations}' \ jq`
IRSA annotation kubectl describe sa ebs-csi-controller-sa -n kube-system Has eks.amazonaws.com/role-arn annotation
IAM role policies aws iam list-attached-role-policies Contains AmazonEBSCSIDriverPolicy
IAM simulation aws iam simulate-principal-policy Shows "EvalDecision": "allowed"

Top comments (0)