Ankit Anand ✨ for SigNoz

Posted on Oct 31, 2023 • Originally published at signoz.io

Complete Guide on Docker Logs [All access methods included]

#devops #logs #docker

This article was originally posted on SigNoz Blog and is written by Favour Daniel.

Docker logs play a critical role in the management and maintenance of containerized applications. They provide valuable information about the performance and behavior of containers, allowing developers and administrators to troubleshoot issues, monitor resource usage, and optimize application performance. By capturing and analyzing log data, organizations can improve the reliability, security, and efficiency of their containerized environments.

In this article, we will explore the concept of Docker logs, the types, and how they can be accessed and effectively managed for optimal use in a containerized environment.

A brief overview of Docker

Docker is a cutting-edge platform for building, distributing, and operating distributed applications. With this technology, developers are able to package their applications and dependencies into containers. These containers can run on any system equipped with a Docker engine.

Containers are lightweight and portable software packages that contain everything needed to run the application, including the code, runtime, system tools, libraries, and settings. This containerization process allows for seamless deployment and scaling of applications across various environments, leading to consistency and reproducibility in the development process.

Throughout the development of an application, different components might require different operating systems or configurations. By using Docker, developers can create containers for each component and specify necessary dependencies and configurations. This way, the application can function consistently across different environments, making it easier to test, debug, and deploy. Additionally, since containers are isolated from one another, changes made to a single container will not affect other components or the host operating system. This is the basic concept behind how Docker and containers work.

Now let us understand what docker logs are.

What are Docker Logs?

Docker logs refer to the records of events and messages generated by a Docker container or Docker engine. These logs provide insight into the activities and operations of a container, including its start and stop events, output messages, and error messages.

The collection and analysis of Docker logs play an essential role in monitoring, troubleshooting, and maintaining the stability and performance of Docker containers.

Types of Docker Logs

In this section, we will delve into the various types of Docker logs that are generated as a result of container activity. They are:

Container Logs

Container logs are records of the standard output and error streams generated by a containerized application. They contain any messages or errors produced by the application as it runs and can be used for troubleshooting and monitoring purposes.

Logs can be collected through various methods and viewed using tools such as the docker logs command or a centralized logging platform like SigNoz. These logs are essential in ensuring the proper functioning and performance of the application.

Daemon Logs

Daemon logs are records of events and messages generated by background processes known as Daemons. These processes perform system-level tasks, such as managing network connections and services or executing scheduled tasks. These logs contain information about the daemon's activities, including status updates, error messages, and performance metrics, and are useful for debugging, monitoring, and auditing purposes.

The logs are typically stored in text files and can be viewed and analyzed using tools like the system's log viewer or a centralized logging platform. These logs play an essential role in ensuring the proper functioning and stability of a computer's background processes.

Methods of Accessing Docker Logs

Gaining insight into the activity and performance of your Docker containers is essential for ensuring the smooth operation of your applications. In this section, we will explore the various methods available for accessing Docker logs, which include:

Accessing Docker logs with Docker CLI

The Docker Command Line Interface (CLI) provides a means for users to interact with Docker components, including containers, images, networks, and more. To access logs generated by Docker containers, the docker logs command can be utilized.

The syntax for accessing docker logs with the CLI:

docker logs [OPTIONS] CONTAINER

where:

CONTAINER is the name or ID of the container that you want to view the logs of.
OPTIONS is an optional flag that you can use to specify the details of the logs that you want to retrieve. To explore different options available for usage, refer to Docker's official documentation.

The docker logs command retrieves and displays the logs generated by a container in the console. The logs can be viewed in real-time or after the container has stopped. By default, the command retrieves all logs produced by the container, however, the user has the option to specify a time range or limit the number of logs displayed. This provides a flexible approach to accessing and reviewing the logs of a container. The docker logs command is a valuable tool for debugging and monitoring the performance of Docker containers.

Accessing Docker Logs with Docker API

The Docker Application Programming Interface (API) enables developers to access and manage Docker components programmatically, including containers, images, networks, and more. The API also provides access to logs generated by Docker containers. A variety of options are available for retrieving logs through the API, including the ability to retrieve logs for a specific container, view logs in real time, and limit the logs to a specific time range or a number of lines. We will look at these options now.

Retrieving logs from a specific container:

We can use an API endpoint to retrieve logs from a Docker container. Here is a basic example using curl:

First, obtain the container ID using the below command;

$ docker ps

Next, retrieve the logs from the container using the API endpoint, as shown below:

$ curl --unix-socket /var/run/docker.sock http:/containers/<CONTAINER_ID>/logs?stderr=1&stdout=1

Note: You will need to replace <CONTAINER_ID> with the actual ID of the container you want to retrieve logs from.

In the above example, the stderr and stdout parameters are set to 1 to retrieve both standard output and standard error logs. If you only want to retrieve logs from one of these sources, you can set the corresponding parameter to 0.

View logs in real-time:

The following example shows how to retrieve logs in real-time using the Docker API. The follow query parameter is set to true to enable real-time logs.

$ curl --unix-socket /var/run/docker.sock <http://v1.40/containers/container_id/logs?stdout=true&stderr=true&follow=true>

Limiting the logs to a specific time range or number of lines:

The following example shows how to limit the logs to a specific time range using the Docker API. The since query parameter is used to specify the start time, while the until parameter is used to specify the end time.

$ curl --unix-socket /var/run/docker.sock <http://v1.40/containers/container_id/logs?stdout=true&stderr=true&since=2022-01-01T00:00:00Z&until=2022-12-31T23:59:59Z>

Similarly, the following example shows how to limit the logs to a specific number of lines using the tail query parameter.

$ curl --unix-socket /var/run/docker.sock <http://v1.40/containers/container_id/logs?stdout=true&stderr=true&tail=100>

Logging Drivers

Logging drivers are plugins in the Docker ecosystem that provide a means to redirect logs generated by Docker containers to various log storage destinations, such as local or remote files, centralized log servers, or cloud-based logging services. The logging driver is configured on the Docker daemon and determines the method used to collect and store logs from containers.

Docker has several built-in logging drivers, including JSON files (default logging driver), Syslog, Journald, and Fluentd, each with its advantages and disadvantages. You can check out more logging drivers available. Learn how to configure a Docker daemon to a logging driver from this guide.

The JSON File logging driver, for example, stores logs as JSON objects in a local file, which is useful for debugging, while the Syslog logging driver forwards logs to a remote syslog server for centralized log management. The Journald logging driver sends logs to the local system's journal, and the Fluentd logging driver forwards logs to a Fluentd log collector.

To set up Syslog as your logging driver, refer to this guide.

Collecting Docker logs with a Log Analytics tool

Log analytics tools provide a way to collect, process, and store logs generated by Docker containers. These solutions provide a number of benefits over using built-in logging drivers, including advanced log analysis, centralized log management, and more robust logging capabilities.

Log analytics tools can be used in combination with the Docker API and logging drivers to provide a complete logging solution for Docker containers. For example, logs from containers can be sent to an external log server using a logging driver, and then analyzed and visualized using a logging solution like SigNoz.

Best Practices for Managing Docker Logs

Efficient log management is key to ensuring the performance and reliability of containerized applications. In this section, we will explore various strategies for managing Docker logs. They are:

Establishing Log Retention Policies

A well-defined log retention policy is a cornerstone of effective log management. This policy outlines the length of time that logs will be stored and when they will be deleted. By establishing such a policy, organizations can ensure that logs do not consume excessive disk space and that relevant information is readily available for debugging and analysis purposes. In addition, a retention policy helps to streamline the log management process and enables organizations to retain only the data that is necessary for their specific needs.

Configuring Log Rotation

Log rotation is an essential aspect of log management that helps to organize logs and conserve disk space by periodically moving older logs to a separate file and writing new logs to the current log file.
Effective log rotation configuration requires consideration of factors such as log size, frequency of generation, and retention requirements. For example, some logs may need to be kept for an extended period of time due to compliance or regulatory requirements, while others may have a shorter retention period.
To configure log rotation for your Docker containers, refer to this guide.

Cleaning Logs

Maintaining an organized log infrastructure requires regular cleaning of logs to prevent disk space from being consumed by unnecessary data. This involves identifying and removing logs that have reached their established retention period and are no longer needed.

Cleaning logs also helps to ensure that logs are easily accessible and readable when they are needed for debugging and analysis. If logs are not cleaned regularly, they can become cluttered and difficult to navigate, making it difficult to quickly find the information needed to diagnose issues with containers and applications.

Archiving Logs for Long-Term Storage

Archiving logs for long-term storage is a crucial aspect of log management as it ensures that valuable information is retained for future reference. This may include logs needed for auditing, compliance, and forensic purposes. Archived logs can be stored in a separate server or a cloud storage solution to ensure they are secure and easily accessible.

It is important to follow a well-defined archiving process that includes regularly backing up logs and ensuring their availability and integrity over time. Additionally, it is advisable to have a disaster recovery plan in place to ensure that logs are not lost in case of any unexpected events.

Monitoring Logs

Proactive monitoring of logs is a key aspect in ensuring the stability and efficiency of containerized applications. Through regular inspection of logs, potential problems can be detected early, allowing for prompt resolution.

This can be accomplished by utilizing the Docker CLI, the Docker API, or leveraging external logging solutions. By implementing a consistent log monitoring process, the reliability and performance of containerized applications can be greatly improved.

Docker Logs Analysis with SigNoz

By leveraging a log analytics tool like SigNoz, organizations can benefit from advanced log management capabilities, including real-time log collection, aggregation, and analysis, as well as centralized log storage and retrieval.

SigNoz is a full-stack open-source solution for Application Performance Monitoring that streamlines the process of monitoring logs, metrics, and traces. Log management is a crucial aspect of observability, and SigNoz offers a wide range of tools to help you manage, collect, and analyze logs generated by Docker containers.

The tool leverages the power of ClickHouse, a high-performance columnar database, to store and access log data for efficient analysis. Moreover, SigNoz adopts the latest standard for instrumenting cloud-native applications, OpenTelemetry which is backed by CNCF.

The logs tab in SigNoz is packed with advanced features that streamline the process of analyzing logs. Features such as a log query builder, search across multiple fields, structured table view, and JSON view make the process of analyzing Docker logs easier and more efficient.

SigNoz offers real-time analysis of logs, enabling you to search, filter, and visualize them as they are generated. This can assist in identifying patterns, trends, and problems in the logs and resolving issues efficiently.

With the advanced Log Query Builder, you can filter out logs quickly with a mix and match of fields.

Advanced query builder to search and filter logs quickly in SigNoz dashboard

Getting Started with SigNoz

SigNoz can be installed on macOS or Linux computers in just three steps by using a simple install script.

The install script automatically installs Docker Engine on Linux. However, on macOS, you must manually install Docker Engine before running the install script.

git clone -b main <https://github.com/SigNoz/signoz.git>
cd signoz/deploy/
./install.sh

You can visit our documentation for instructions on how to install SigNoz using Docker Swarm and Helm Charts.

If you liked what you read, then check out our GitHub repo 👇

Conclusion

Docker logs are critical components of managing and maintaining the health of Docker containers and their applications. By leveraging the power of Docker logs, organizations can optimize their container-based infrastructure and improve their ability to troubleshoot, analyze, and monitor the performance of their applications.

It is important to have a robust and efficient log management strategy in place, as it helps to ensure that logs are captured, stored, and analyzed effectively. Adopting a dedicated log management tool, instead of relying solely on the native methods of accessing and managing Docker logs, can provide a range of advanced features and greater flexibility for analyzing and processing logs from your containers.

Log management tools like SigNoz can enhance the capability to handle logs generated by Docker containers. It offers a comprehensive and scalable approach to log analytics that can cater to specific needs and requirements, which might not be met by the native Docker logging options like logging drivers or the Docker API. For instance, advanced log parsing, filtering, or transforming functions that are not feasible using just the logging drivers or the Docker API can be carried out with SigNoz.

Configure your Docker Syslog Logging Driver | SigNoz

DEV Community