Flow Logs is a unique feature that enables you to capture traffic inbound and outbound from your AWS network interfaces. There are three types of flow logs:
a. VPC flow logs
b. Subnet Flow Logs
c. Elastic Network Interface Flow logs.
Our focus will be on VPC flow logs. As the name implies, we will capture traffic entering and leaving VPC interfaces. VPC flow logs can be stored in either Cloudwatch logs or Amazon S3.
The logs that will be captured for this scenario will be stored in Amazon S3 and queryable on the Anthena platform for simplified output.
Kindly follow the steps below:
STEP 1
Create the VPC flow logs on the existing VPC
STEP 2
Give the flowlog a name and specify the destination where the logs will be stored. In our case, we are storing the logs captured inside the S3 bucket. For this reason,specify the S3 bucket ARN where you want to store your log.
STEP 3
Log files have been stored in an S3 bucket.
STEP 4
Create a work group on Anthena.
https://docs.aws.amazon.com/athena/latest/ug/vpc-flow-logs.html
STEP 5.
Create a table in the default database.
CREATE EXTERNAL TABLE IF NOT EXISTS vpc_flow_logs (
version int,
account_id string,
interface_id string,
srcaddr string,
dstaddr string,
srcport int,
dstport int,
protocol bigint,
packets bigint,
bytes bigint,
start bigint,
end bigint,
action string,
log_status string,
vpc_id string,
subnet_id string,
instance_id string,
tcp_flags int,
type string,
pkt_srcaddr string,
pkt_dstaddr string,
region string,
az_id string,
sublocation_type string,
sublocation_id string,
pkt_src_aws_service string,
pkt_dst_aws_service string,
flow_direction string,
traffic_path int
)
PARTITIONED BY (date date)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ' '
LOCATION 's3://s3flowlog1/AWSLogs/003985890001/vpcflowlogs/us-east-1/'
TBLPROPERTIES ("skip.header.line.count"="1");
STEP 6.
Alter the table and add a partition.
ALTER TABLE vpc_flow_logs
ADD PARTITION (date='2023-05-11')
LOCATION 's3://s3flowlog1/AWSLogs/003985890001/vpcflowlogs/us-east-1/2023/05/131';
STEP 7
Query the database and analyze your output.
a.
SELECT * FROM vpc_flow_logs .
b.
SELECT
interface_id,
srcaddr,
action,
protocol
FROM vpc_flow_logs
WHERE action = 'REJECT' AND protocol = 6
LIMIT 10
c.
SELECT
interface_id,
srcaddr,
action,
protocol
FROM vpc_flow_logs
WHERE action = 'REJECT' AND protocol = 6
Top comments (1)
Anthena is a useful tool for querying logs. Your article explains in details about VPC flow logs. Very good work.