Amazon EKS Fargate now supports a built-in Fluent Bit-based log router, eliminating the need for running sidecar containers to collect and ship logs. This simplifies log management by reducing overhead and complexity while providing an efficient way to ship logs from EKS pods to Amazon CloudWatch.
This article will guide you through configuring Fluent Bit on EKS Fargate to ship logs to CloudWatch, both globally for all pods and specifically for individual deployments.
Built-in Logging with Fluent Bit
EKS Fargate uses Fluent Bit for log routing. By applying a ConfigMap in the aws-observability namespace, you can configure log routing for all pods in the cluster. The logging ConfigMap affects applications cluster-wide, regardless of their namespace.
Step 1 : Grant Required IAM Permissions
Ensure the Fargate pod execution role includes the following permissions to allow Fluent Bit to send logs to CloudWatch:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:CreateLogGroup",
"logs:DescribeLogStreams",
"logs:PutLogEvents",
"logs:PutRetentionPolicy"
],
"Resource": "*"
}]
}
Step 2 : Create the aws-observability Namespace
The Fluent Bit configuration must reside in the aws-observabilitynamespace.
apiVersion: v1
kind: Namespace
metadata:
name: aws-observability
labels:
aws-observability: enabled
Apply this namespace using:
kubectl apply -f namespace.yaml
Step 3 : Configure Fluent Bit
Send All Logs to One CloudWatch Log Group
To route all logs to a single CloudWatch log group, create the following ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-logging
namespace: aws-observability
data:
output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match *
region <region-code>
log_group_name fluent-bit-cloudwatch
log_stream_prefix from-fluent-bit-
auto_create_group true
Use Dedicated Log Groups for Each Deployment
If you wish to route logs to specific CloudWatch log groups per deployment, use a more detailed ConfigMap:
kind: ConfigMap
apiVersion: v1
metadata:
name: aws-logging
namespace: aws-observability
data:
output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match kube.var.log.containers.deployment1*
region eu-west-1
log_group_name fluent-bit-cloudwatch-deployment1
log_stream_prefix from-fluent-bit-
auto_create_group true
[OUTPUT]
Name cloudwatch_logs
Match kube.var.log.containers.deployment2*
region eu-west-1
log_group_name fluent-bit-cloudwatch-deployment2
log_stream_prefix from-fluent-bit-
auto_create_group true
filters.conf: |
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude On
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
#Replace the "deployment1" with your deployment name, and re-use the output conf as per the number of deployments you have.
Step 4 : Restart Fargate Deployments
After applying the ConfigMap, restart your Fargate pods to apply the new logging configuration:
Validation
Check if Logging is Enabled
To verify that logging is enabled for a pod, describe the pod using kubectl:
kubectl -n <namespace> describe po <pod-name>
Look for the following annotation:
Annotations: Logging: LoggingEnabled
Verify Logs in CloudWatch:
- Open the CloudWatch console.
- Navigate to Log groups.
- Confirm that the logs are being shipped to the configured log groups.
That’s it. Thank you for taking the time to read this article! Keep up the great work, and happy deploying! 🚀 😊
Top comments (0)