DEV Community

Ashish Gajjar
Ashish Gajjar

Posted on

How to Send Log Files to AWS CloudWatch Logs

AWS CloudWatch allows you to collect logs from your AWS EC2 instances. Files such as the audit, access and error logs that are commonly found on web servers. Or any /var/log file. This is specially useful if you have a instances behind a load balancer. Rather than connecting to each instance and manually searching the logs with grep, CloudWatch centralizes the logs into one log stream, allowing you to search all your log files from one place.

For example, we have a few EC2 instances behind a Load Balancer. we send the contents of the log to CloudWatch. Even with just a few servers it’s much easier than logging into each one individually and searching each file with grep commands. Our User Journey infrastructure has over 100 servers and would be practically impossible to manage through the command line alone. If an issue occurs we’re able to see all the logs in the AWS Console without wasting time logging onto everything.

CloudWatch set-up on EC2-instance , you need to complete the following:

  1. Create a new IAM role (one time only)
  2. Attach the IAM role to an EC2 instance
  3. Install and configure the awslogs service in EC2-Machine.
  4. View logs in CloudWatch Logs groups.

1. Create a New IAM Role

To allow an EC2 instance to communicate with CloudWatch, you first need to create an IAM Role. You only need to do this once.

  • Open AWS Console and open the IAM console. .
    Image description

  • From the AWS menu, select Roles and then click the Create role button. Choose the service that will use this role, select EC2 and click the Next: Permissions button:
    Image description

  • Search for the CloudWatchFullAccess, check the checkbox and click Next:
    Image description

  • Enter a Role name (e.g. Cloudwatchlogs). Then click Create role.
    Image description

2. Attach the IAM Role

To attach the IAM Role to the EC2 instance, you can either do it through the AWS console.

  • Using the AWS Console Go to the EC2 Dashboard, select Instances from the menu and check the checkbox next to the EC2 instance you want to stream the logs from. To attach the IAM Role, click the Actions dropdown and select Security > Attach/Replace IAM Role:

Image description

  • Search for and select the IAM role created above (e.g. CloudWatchlogs), then click Apply to attach the IAM role:

Image description

  • Verify IAM Role in Security Image description

3. Install AWS logs service in EC2-Instance

  • ssh ec2-instace and install awslog service
    Image description

  • Edit the /etc/awslogs/awslogs.conf file to configure the logs to track

[ec2-user@ip-172-31-95-1 ~]$ sudo vim /etc/awslogs/awscli.conf
[plugins]
cwlogs = cwlogs
[default]
region = us-east-1
Enter fullscreen mode Exit fullscreen mode
  • Start the awslogs service.
[ec2-user@ip-172-31-95-1 ~]$ sudo  systemctl enable --now awslogsd.service
Enter fullscreen mode Exit fullscreen mode

4. View logs in CloudWatch Logs groups.

  • Once the log file you are watching has data written to it, you’ll be able to find it in CloudWatch. Go to the CloudWatch Overview and select Logs from the menu. You should see the /var/log/message. Image description
  • Click on the log group name to see the log streams. Each log stream uses the EC2 instance ID, so you know which EC2 instance logged the data: Image description
  • To search the logs, click the Search Log Group button. In the filter text box, enter a search term to search all your log files in one go: Image description

Pricing:
Free Tier:
EC2 : 750 hrs.
CloudWatch Logs: 5 GB Data (ingestion, archive storage, and data scanned by Logs Insights queries).

Top comments (0)