Fixing Fluent Bit on EKS: Solving the "NoCredentialProviders" Error

Fixing Fluent Bit on EKS: IRSA, Helm, and the "NoCredentialProviders" Error

If you're deploying Fluent Bit to Amazon EKS with Terraform and Helm, you might hit this frustrating error:

[cloudwatch 0] NoCredentialProviders: no valid providers in chain
...
EC2MetadataError: failed to make EC2Metadata request

Even if everything looks correct — the service account is annotated with an IAM role, and the DaemonSet is running — Fluent Bit still can’t write logs to CloudWatch.

Let's walk through how to troubleshoot and fix this.

🔍 Step 1: Understand the Root Cause

Fluent Bit uses AWS credentials to ship logs to CloudWatch. These credentials usually come from:

• IRSA (IAM Roles for Service Accounts)
• EC2/EKS metadata (instance profile)
• Environment variables

When using the aws-for-fluent-bit Helm chart, if you don’t explicitly configure the service account, Helm will create a new one without any IAM role annotation.

You can confirm this with:

kubectl get pod -n amazon-cloudwatch -l app.kubernetes.io/instance=<your-release-name> -o jsonpath="{.items[*].spec.serviceAccountName}"

If it returns something like:

fluent-bit-aws-for-fluent-bit

Instead of your IRSA-linked service account name (e.g., fluent-bit), Fluent Bit is running without credentials.

✅ Step 2: Ensure a Proper IRSA Setup

Make sure the IRSA role exists and is attached to a manually created Kubernetes service account:

resource "kubernetes_service_account" "fluentbit" {
  metadata {
    name      = "fluent-bit"
    namespace = "amazon-cloudwatch"
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.fluentbit_irsa.arn
    }
  }
}

And the IAM role should have at least these permissions:

resource "aws_iam_policy" "fluentbit_logs" {
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Action = [
          "logs:PutLogEvents",
          "logs:DescribeLogStreams",
          "logs:DescribeLogGroups",
          "logs:CreateLogStream",
          "logs:CreateLogGroup"
        ],
        Resource = "*"
      }
    ]
  })
}

Attach it to the IRSA role.

✅ Step 3: Update the Helm Release in Terraform

In your helm_release resource, add the following set blocks:

set {
  name  = "serviceAccount.name"
  value = "fluent-bit"
}

set {
  name  = "serviceAccount.create"
  value = "false"
}

That tells Helm to use your IRSA-linked SA and not to create its own.

Here’s a full example:

resource "helm_release" "fluentbit" {
  name       = "fluent-bit"
  namespace  = "amazon-cloudwatch"
  repository = "https://aws.github.io/eks-charts"
  chart      = "aws-for-fluent-bit"
  version    = "0.1.35"

  set {
    name  = "serviceAccount.name"
    value = "fluent-bit"
  }

  set {
    name  = "serviceAccount.create"
    value = "false"
  }

  set {
    name  = "cloudWatch.enabled"
    value = "true"
  }

  set {
    name  = "cloudWatch.region"
    value = data.aws_region.current.name
  }

  set {
    name  = "cloudWatch.logGroupName"
    value = "/aws/eks/${var.cluster_name}/fluent-bit"
  }

  set {
    name  = "cloudWatch.autoCreateGroup"
    value = "false"
  }
}

🔄 Step 4: Apply and Restart

Apply the changes:

terraform apply

Then restart the Fluent Bit pod:

kubectl delete pod -n amazon-cloudwatch -l app.kubernetes.io/instance=fluent-bit

Check logs:

kubectl logs -n amazon-cloudwatch -l app.kubernetes.io/instance=fluent-bit

No more NoCredentialProviders 🎉

✅ Recap

✔️ Confirm you're using an IRSA-annotated service account
✔️ Tell Helm to use it with serviceAccount.name and create=false
✔️ Give the IAM role the correct CloudWatch permissions
✔️ Restart pods to pick up changes

Now Fluent Bit can log to CloudWatch securely using IAM — no access keys needed.

Ready to take this up a notch? Turn it into a Terraform module or plug it straight into your CI/CD and forget it ever broke in the first place. 😎