Anatomy Of Cloudwatch Logs

Introduction

Cloudwatch logs is one of the services offered by AWS, under the banner of cloudwatch. As the the name suggests, it is used for monitoring and storing logs. These logs can be generated from different services/resources such as ECS, Lambda, Batch Jobs (EC2 in general) et cetra. In this post we'll take a look at the anatomy of cloudwatch logs, and some code samples for fetching these logs.

Anatomy

There are 3 main terminologies associated with the service:

1. Log Group : The user guide defines log group as A log group is a group of log streams that share the same retention, monitoring, and access control settings. In order to grokk it easily, you can also think of a log group as a collection of all
   logs that belongs to a particular service. For instance, you can have a log group for ecs service, a log group for a lambda and so on. In most cases AWS will create log groups for you by default.

2. Log Stream : Log streams are a sequence of logs event (defined below) that are coming from the same source. For instance, think of a lambda execution, all the logs that are created by one full execution of a lambda will end up in one log stream.
   Similarly all the logs produced by one full execution of a batch job will also end up in one log stream.

3. Log Event: Every single piece of logging, every Logger.info(), every System.out.println(), every print() constitutes one log event. Log event has 3 propertes:
   a) Message (the actual message)
   b) Timestamp (when the message was generated)
   c) Ingestion Timestamp (when the message was ingested to cloudwatch)

The following image show the relation between log group, log streams, and log event

Using The Java SDK

In this part we'll take a look at retrieving all the logs from a particular log stream. This could be useful if you want to automate some log processing, say in the event of a lambda failure.s

• Maven Dependency (version mentioner mentioned is for symbolic purpose, make sure you pick the latest version or the one compatible with your existing dependencies)

    <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-logs</artifactId>
            <version>1.11.519</version>
        </dependency>

• Create A Logs Client (Default)

AWSLogs logsClient = AWSLogsClientBuilder.defaultClient();

• Retrieving Logs First we need to create a request object. To start off, we'll pass log group name and log stream name

GetLogEventsRequest logEventsRequest = new GetLogEventsRequest(logGroupName, logStreamName)

Now we can use the client we created above to get the logs :

GetLogEventsResult logEventsResult = logsClient.getLogEvents(logEventsRequest);

GetLogEventResult contains following info:

1. list of log events List<OutputLogEvents>
2. next forward token
3. next backward token

What's with the tokens? When we make a call to get log events, the result may not contain all the log; there's a limit of 1 MB (10000 log events). If no more logs are being being written in the particular log stream then you'll have mostly have old log events that needs to be fetched by using next backward token. If there are new log events in the log stream, then using next forward token will yield more log events. In such case, its better to assume that the first result we get lands us in the middle of logs and that we need to traverse backward and forward in order to get the complete picture.

Visual Representation Of Using Tokens To Fetch More Logs

We can do this in iteratively or recursively. Below is code sample for using the tokens iteratively:

        List<OutputLogEvent> outputLogEvents = new LinkedList<>();
        GetLogEventsResult eventsResult = new GetLogEventsResult();

        String nextToken = token;

        do{
            logEventsRequest.setNextToken(nextToken);
            eventsResult = logsClient.getLogEvents(logEventsRequest);
            outputLogEvents.addAll(eventsResult.getEvents());

            nextToken = tokenDirection.equals(TOKEN_DIRECTION.BACKWARD) ? eventsResult.getNextBackwardToken() 
                    : eventsResult.getNextForwardToken();

            LOGGER.info("Next Token: " + nextToken);

        }while (CollectionUtils.isNotEmpty(eventsResult.getEvents()));

You can be mindful about the sequence while traversing, or can just fetch all the event and sort them using the
timestamp.

    outputLogEvents
                .stream()
                .sorted(Comparator.comparing(OutputLogEvent::getTimestamp))
                .collect(Collectors.toList());

With that, the article comes to an end.
Cheers!

Further Reading & Reference Used

a- Cloudwatch Logs USer Guide

b- SDK Javadoc