Guide on Structured Logs [Best Practices included]

This article was originally posted on SigNoz Blog and is written by Joseph Chege.

Structured logging is the method of having a consistent log format for your application logs so that they can be easily searched and analyzed. Having structured logs allows for more efficient searching, filtering, and aggregation of log data. It enables users to extract meaningful information from log data easily.

Logging is an essential aspect of system administration and monitoring. Logging allows you to record information data about the application's activity. The primary purpose of obtaining logs is to streamline the debugging, troubleshooting, or auditing practices.

Logs include system events, user actions, and error messages. They can be stored in various forms, such as text files and databases. This allows them to analyze and visualize using log management platforms such as SigNoz. The structure of these logs falls on a large spectrum. A log can be structured, semi-structured or unstructured. In this blog, we will learn about structured logging and its benefits.

What are Structured Logs?

A structured log is a log record with a well-defined structure and format. The structured logging practice creates easily human-readable textual log data. The same data can be easily integrated with other systems to automate log analyzing, querying, and aggregation.

Semi-structured and unstructured logging usually provide text data that are easy to read by humans but can be challenging for machines to extract. It is harder to query such data structures for helpful information. Unstructured log data makes it harder to analyze log data.

Structured logging solves these challenges and creates room for additional analytic techniques. Structure logging aims to create machine-readable data for advanced functionalities. Structured logs ensure each log entry is composed of a set of fields or key-value pairs in a well-structured and consistent format that can be easily parsed and analyzed.

This structure allows for more efficient searching, filtering, and aggregation of log data. With the help of logging tools, you can automatically extract meaningful information from the logs. It becomes more straightforward to transform your data efficiently. Due to their format, structured logs also allow effortless integration with other systems, such as monitoring and alerting tools.

Consider the following simple example of an unstructured log message:

[info] [Friday, 20-Jan-23 11:17:55 UTC] The application has started.
[Error] [Friday, 20-Jan-23 11:17:55 UTC] An error occurred while processing this request.

The above log example contains log information. While it is a straightforward log, having a huge list of such records that includes complex examples can take time to draw a valid conclusion. Imagine having thousands of such logs that you need to analyze for insights.

It can be hard to diagnose and troubleshoot how an issue occurred, as the log message needs descriptive log information. Using the same log message, you can use structured log formats to describe the message with payload:

{
  "timestamp": "Friday, 20-Jan-23 11:17:55 UTC",
  "level": "info",
  "message": "The application has started."
}

Both the unstructured and structured payloads contain the same information. However, the structured log message has key attributes that can be paired with any corresponding values. This means an analysis system can analyze the data using these attributes to perform searches and filter results. On top of that, you can detect patterns in the data and have a concrete conclusion based on your log messages.

With key-value pairs, you can add more meaningful information to your logs. Here is an example of log using a structured format with more details on error type:

{
  "timestamp": "Friday, 20-Jan-23 11:17:55 UTC",
  "level": "error",
  "message": "An error occurred while processing this request",
  "error": {
    "code": "500",
    "message": "Internal server error",
    "details": "Error occurred when sending POST data query."
  }
}

Why is Structured Logging Needed?

There are numerous benefits to using structured logs. Below are some points for why structured logging is needed:

• With structured logs, it is accessible to search, filter, and aggregate log data.
• Its format is friendly with observability and logging tools like SigNoz. This makes it easier to analyze your log data faster and more efficiently to troubleshoot any application issues.
• Structured logs improve automation. You can automatically extract any relevant information, such as error messages and performance metrics, and create aggregates.
• Structured logs are easily integrated with alert systems. You can get notified if the software system starts sending logs of a certain type.
• With structured logs, it is easier to identify trends and patterns that might be missed with unstructured logs.
• Structured logs open room for broader integration with logs from different sources and with monitoring tools.
• Structured logs reduce costs on tools you can choose to integrate and analyze data. The data is consistent. This makes storing in long-term retention easier with reduced storage cost.
• It is easier to visualize structured logs data with which you can drive insights faster. Log analytics tool like SigNoz can help you create charts from structured logs data easily.

Best Practices for Structured Logging

Structured logging has many benefits. Based on this guide, you have understood the importance in detail. However, when using structured logs, it's good to follow the best practices to ensure you get the optimal benefits structured logs offer. Some of the best practices are:

• Always use a consistent format if you choose JSON, for example. You should stick to that one format across your application. This makes it easier to parse the collected data, analyze logs and provide timely feedback.
• Remember to include relevant information in your log entries. Fields such as timestamps, logging levels, and context-specific data (user information and request details) make it easier to understand your application history.
• Use the standard log levels for your log entries. This includes info, warning, error, debugging, etc. Such practices make it possible to have filtering features for your log data.
• Remember to add unique keys in log entries. This makes it easy to search and filter log data using tools like SigNoz.
• Create security-logging practices to monitor security breaches, suspicious activity, and other potential threats.
• Before rolling your application to production, always test and validate your logs. This will help you ensure your application generates logs correctly and in the expected format.
• You can also integrate your logs with traces. Having correlated telemetry signals can help you drive contextual insights faster.

Getting Started with Structured Logging

The first step to structured logging starts at the application level. There are numerous logging libraries that help with structured logging. For example, winston logger is one of the most popular logging libraries for nodejs. It provides consistent formats and logging levels to be used in application logs.

The next step is to choose a log analytics tool. When choosing a log analytics tool, three things must be kept in mind - ingestion, query, and storage. Based on these parameters and your use case, you can choose a log analytics tool that suits you. SigNoz is an open-source log management tool that you can use for structured logging.

It uses OpenTelemetry to collect logs and uses a columnar database for storage. You can check out the logs performance benchmark of SigNoz to see if it fits your needs.

It is easy to get started with SigNoz. SigNoz can be installed on macOS or Linux computers in just three steps by using a simple install script.

The install script automatically installs Docker Engine on Linux. However, on macOS, you must manually install Docker Engine before running the install script.

git clone -b main https://github.com/SigNoz/signoz.git
cd signoz/deploy/
./install.sh

You can visit our documentation for instructions on how to install SigNoz using Docker Swarm and Helm Charts.

Conclusion

Logging is an integral part of all software systems. You can understand what's happening and where problems might be. Generally, if something goes wrong in production, you have a reference to trace your logs. Structured logs make it easier to search, query, and analyze logs data at scale.

A log analysis tool like SigNoz can help you derive insights from logs data quickly while ensuring efficient management and storage. You can check out the SigNoz GitHub repo here.

Related Posts

A Lightweight Open Source ELK alternative

OpenTelemetry Logs - A Complete Introduction & Implementation