DEV Community

Yen Trinh
Yen Trinh

Posted on • Updated on

How to query AWS load balancer log if there are terabytes of logs?

I want to query AWS load balancer log to automatically and on schedule send report for me.

I am using Amazon Athena and AWS Lambda to trigger Athena. I created data table based on guide here:

https://docs.aws.amazon.com/athena/latest/ug/application-load-balancer-logs.html

However, I encounter following issues:

1) Logs bucket increases in size day by day. And I notice if Athena query need more than 5 minutes to return result, sometimes, it produce "unknown error"

2) Because the maximum timeout for AWS Lambda function is 15 minutes only. Therefore, I can not continue to increase Lambda function timeout to wait for Athena to return result (if in the case that Athena needs >15 minutes to return result, for example)

Can you guys suggest for me some better solution to solve my problem? I am thinking of using ELK stack but I have no experience in working with ELK, can you show me the advantages and disadvantages of ELK compared to the combo: AWS Lambda + AWS Athena? Thank you!

Top comments (0)