Docker Logging - A Complete Guide to Logs in Docker

This tutorial was originally posted on SigNoz Blog and is written by Muskan Paliwal

Log analysis is a very powerful feature for an application when it comes to debugging and finding out which flow is working properly in the application and which is not. Log management helps the DevOps team debug and troubleshoot issues faster, making it easier to identify patterns, spot bugs, and ensure they don’t come back to bite you!

In this article, we will discuss log analysis in Docker and how logging in Docker containers is different than in other applications. These logs are specific to Docker and are stored on the Docker host. We’ll thoroughly discuss the docker logs command and how we can configure a logging driver for containers.

Why is Docker logging different?

Life would be much simpler if applications running inside containers always behaved correctly. But unfortunately, as every developer knows, that is never the case.

With other systems, recording application log messages can be done explicitly by writing those messages to the system logger. This is done using syslog() system call from our application. But this doesn’t work for containerized logging; here’s why:

• Containers are Multi-leveled
  Containers are just like boxes inside a big box. No matter how simple the Docker installation is, we’ll have to deal with two levels of aggregation. One level is where you see logs inside the container in your Dockerized application, known as Docker Container Logs.

  The second level where you see the logs from the host servers (that is, system logs or Docker daemon logs). These are generally located in /var/log.
  

  A log aggregator that has access to the host application can not pull log files from the Dockerized application as if they are the host log files. In these scenarios, we will have to find a way to correlate the logs.

• Containers are ephemeral
  Container-based environments keep changing very often but it doesn’t serve well to the monitor. Docker containers emit logs to stdout and stderr output streams.

  Logs are often stored on the Docker host because containers are stateless (failing to remember or save data from previous actions).

  json-file is the default logging driver which writes JSON-formatted logs to a container-specific file.The json-file is stored in the /var/lib/docker/containers/ directory on a Linux Docker host. Here’s how you can access the file:
  

  /var/lib/docker/containers/<container id>/<container-id>-json.log      
  

  It is dangerous to store logs in a Docker host because Docker doesn’t impose any size limit on log files, and they can build up over time and eat into your disk space. It is advised to store logs in a centralized location and enable log rotation for all the Docker containers.

docker logs command

docker logs command is used to get all the information logged by a running container. The docker service logs command is used to do the same by all the containers participating in a service.

The example below shows JSON logs created by the hello-world Docker image using json-file driver:

{"log":"Hello there!\n","stream":"stdout","time":"2022-07-28T22:51:31.549390877Z"}
{"log":"This message shows that everything seems to be working correctly.\n","stream":"stdout","time":"2022-07-28T22:51:31.549396749Z"}

The log follows a pattern of printing:

• Log’s origin
• Either stdout or stderr
• A timestamp

In order to review a container’s logs from the command line, you can use the docker logs <container-id> command. Using this command, the logs shown above are displayed this way:

Hello there!
This message shows that everything seems to be working correctly.

Here are a few options in the command that you can use to modify the output of your log:

docker logs [OPTIONS] <container-id>

• Using -f or --follow option, if you want to follow the logs:
  

  docker logs <container_id> --follow
  

• If you want to see the last N log lines:
  

  docker logs <container-id> --tail N
  

• If you want to see the specific logs, use the grep command:
  

  docker logs <container_id> | grep pattern
  

• If you want to show errors:
  

  docker logs <container_id> | grep -i error
  

Once an application starts growing, you tend to start using Docker Compose. docker-compose logs command shows logs from all the services running in the containerized application.

Note that the offering from the docker logs command may vary based on the Docker version you are using. In case of Docker Community, docker logs can only read logs created by the json-file, local, and journald drivers whereas in case of Docker Enterprise, docker logs can read logs created by any logging driver.

Configure a Docker container to use a logging driver

Step1: Configure the Docker daemon to a logging driver

Set the value of the log-driver key to the name of the logging driver in the daemon.json configuration file. Then restart Docker for the changes to take effect for all the newly created containers. All the existing containers will remain as they are.

Let’s set up a default driver with some additional information:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "20m",
    "max-file": "10",
  }
}

To find the current logging driver for the Docker daemon:

$ docker info --format '{{.LoggingDriver}}'

json-file

In order to use a different driver, you can override the default by adding the --log-driver option to the docker run command that creates a container.

For example, the following command creates an Apache httpd container, overriding the default logging driver to use the journald driver instead.

docker run --log-driver journald httpd

Step 2: Deciding the delivery mode of log messages from container to log driver

Docker provides two types of delivery modes for log messages.

• Blocking(default mode)
  As the name suggests, this mode blocks the main process inside a container to deliver log messages. And this will add latency to the performance of the application. But, it ensures that all the log messages will be successfully delivered to the log driver.

  The default log driver (json-files) logs messages very quickly since it writes to the local file system. Therefore it’s unlikely to cause latency. But drivers like gcplogs and awslogs open a connection to a remote server and are more likely to block and cause latency.

• Non-blocking
  In this mode, the container writes logs to an in-memory ring buffer. This in-memory ring buffer works like a mediator between logging-driver and the container. When the logging-driver isn’t busy processing the logs, the container shares the logs to the driver immediately. But when the driver is busy, these logs are put into the ring-buffer.

  This provides you a safety check that a high volume of logging activity won’t affect the application’s performance running inside the container. But there is a downside. It doesn’t guarantee that all the log messages will be delivered to the logging driver. In cases where log broadcasts are faster than the driver processor, the ring buffer will soon run out of space. As a result, buffered logs are deleted to make space for the next set of incoming logs. The default value for max-buffer-size is 1 MB.

  To change the mode:
  

  # /etc/docker/daemon.json
  {
      "log-driver": "json-file",
      "log-opts": {
          "max-size": "20m",
              "max-file": "10",
          "mode": "non-blocking"
      }
  }
  

  Alternatively, you can set the non-blocking mode on an individual container by using the --log-opt option in the command that creates the container:
  

  docker run --log-opt mode=non-blocking alpine echo hello world
  

The default log driver stores data in a local file, but if you want more features, then you can opt for other log drivers as well, such as logagent, syslog, journald, elf, awslogs, etc.

Logging strategies

Docker logging means logging events of the dockerized application, host OS and the docker service. There are various ways to log events for a docker container.

Some of them are:

• Application logging: In this strategy, the application inside the container can have its own logging framework. The logs can either be stored locally or sent to a centralized location using a log management tool.

• Data volumes: Because containers are stateless, and to avoid losing logs data, you can bind the container’s directory to the host OS directory. Containers can now be terminated or shut down, and access logs from multiple containers. You can run a regular backup in order to prevent data corruption or loss in case of failure.

• Docker logging driver: This type has already been discussed in detail. The configured driver reads the data broadcast by the container’s stdout or stderr streams and writes it to a file on the host machine. You can then send this log data anywhere you want to.

Final Thoughts

Containerization surely provides an easy way to deal with application portability and scalability issues but it does requires maintenance time to time. Container environments are just like a box inside a box, with multiple layers of abstraction. So, it becomes hard to debug in such environments and if performed correctly, log-analysis can be your go-to friend to find out performance related issues.

In this guide, you learned how to configure the Docker logging driver for log analysis in containerized applications, how Docker logging is different from application logging hosted on a physical machine or virtual host, and in detail study of the docker logs command.

There are various logging strategies that you can follow for log analysis. This blog thoroughly discussed the default logging strategy - json-file and the two delivery modes of log messages. Containers being stateless, doesn’t ensure data persistence, hence to prevent data loss, you can use various log management tools.

But logs are just one aspect of getting insights from your software systems. Modern applications are complex distributed systems. For debugging performance issues, you need to make your systems observable. Logs, when combined with metrics and traces form an observability dataset that can help you debug performance issues quickly.

SigNoz, an open source APM can help you monitor your application by collecting all types of telemetry data. It correlates all your telemetry data(logs, metrics, and traces) into a single suite of monitoring. It is built to support OpenTelemetry natively. OpenTelemetry is becoming the world standard for instrumenting cloud-native applications.

You can check out SigNoz GitHub repo:

If you want to read more about SigNoz, check out the following blog:

SigNoz - an open source alternative to DataDog