The Complete Guide to Logging in Django

Applications running in production can’t tell you directly what’s going on under the hood, so you need a way to keep track of their behavior. Knowing what’s going on in your code at any given time benefits you from both a technical and a business standpoint. With this information, engineers and product managers can make better decisions about which systems to repair or how to improve user experience (UX).

The most straightforward way to achieve this is with logging. You can code the program during development to share relevant information while running that could be useful in analysis, debugging, and further troubleshooting.

Django is a popular Python web application framework, used by small and large organizations. Python offers a powerful built-in logging module to log events from applications and libraries. The library is flexible and can be easily customized to fit any application. You can implement it this way:

import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info('Start reading database')
# read database here
records = {'john': 55, 'tom': 66}
logger.debug('Records: %s', records)
logger.info('Updating records ...')
# update records here
logger.info('Finish updating records')

When the program is run, you can see the records on the console:

INFO:__main__:Start reading database
INFO:__main__:Updating records ...
INFO:__main__:Finish updating records

This article will delve deeper into the logging concepts that every developer should know. You’ll learn how to log significant data, how to route the logs, and how to consolidate them for insights into your Django applications. You’ll also learn best practices to follow.

What Are Logs?

Logs are the records of events that occur while running software. They contain information about the application, system performance, or user activities. Loggers are the objects that a developer interacts with to print out the information. They help you tell the program what to log and how to do it.

Adding logging to your code involves three critical steps:

• Choosing what data to output and where in the code to do so
• Choosing how to format the logs
• Choosing where to transmit the logs (e.g., stdout or syslog)

How to Implement Logging

To implement logging in a Django application, you must consider the following factors.

Logging Levels

The Python logging library adds several types of metadata to provide valuable context around the log messages. This can help in diagnosing a problem or analyzing what’s happening inside the code while the application is running. For example, log levels define the severity level of log events, which can be used to segment logs so you get the most relevant log message at any specified time.

You can use log levels to help prioritize log messages. For instance, when you’re developing an application, DEBUG information is most relevant; while you’re running the application, you can leave INFO logs to indicate certain events.

The Python logging package comes with five logging levels: critical, error, warning, info, and debug. These levels are denoted by constants with the same name: logging.CRITICAL, logging.ERROR, logging.WARNING, logging.INFO, and logging.DEBUG, with values of 50, 40, 30, 20, and 10. A level’s meaning is determined at runtime by its value.

The community-wide applicability rules for logging levels are as follows:

• DEBUG: logging.DEBUG can be used to log detailed information for debugging code in development, such as when the app starts.

• INFO: logging.INFO can be used to log information about the code if it is running as expected, such as when a process starts in the app.

• WARNING: logging.WARNING can be used to report unexpected behavior that could cause a future problem but isn’t impacting the current process of the application, such as when the app detects low memory.

• ERROR: logging.ERROR can be used to report events when the software fails to perform some action, such as when the app fails to save data due to insufficient permissions given to the user.

• CRITICAL: logging.CRITICAL can be used to report serious errors that impact the continued execution of the application, such as when the application fails to store data due to insufficient memory.

The logging.Logger object offers the primary interface to Python’s logging library. These objects include methods for issuing log requests and for querying and modifying their state, as follows:

• Logger.critical(msg, *args, **kwargs)
• Logger.error(msg, *args, **kwargs)
• Logger.debug(msg, *args, **kwargs)
• Logger.info(msg, *args, **kwargs)
• Logger.warn(msg, *args, **kwargs)

In addition, loggers provide the following two options:

• Logger.log(level, msg, *args, **kwargs) sends log requests with defined logging levels. When you’re using custom logging levels, this approach comes in handy.

• Logger.exception(msg, *args, **kwargs) sends log requests with the logging level ERROR and includes the current exception in the log entries. This function should be called from an exception handler.

Logging Handlers

Logging handlers are used to determine where to put the logs—system logs or files. Unless explicitly specified, the logging library uses a StreamHandler for sending the log messages to the console or sys.stderr.

Handlers also format log records into log entries using their formatters. The formatter for a handler can be set by clients using the Handler.setFormatter(formatter) function. If a handler doesn’t have a formatter, the library’s default formatter is used.

The logging.handler module includes fifteen useful handlers that span a range of use cases (including the ones mentioned above). The most commonly used logging handlers are as follows:

• StreamHandler transmits logs to a stream-like object, such as a console, using stdout.
• FileHandler redirects log events to a file.
• SyslogHandler routes logs to your system’s syslog daemon.
• HTTPHandler allows you to deliver logs through HTTP.
• NullHandler redirects your logs to nowhere, which is helpful for temporarily halting logging.

Logging Formatters

Log messages from the logging library follow this default format:

<LEVEL>:<LOGGER_NAME>:<MESSAGE>

However, they can be customized to add more information using logging.Formatter objects to change them into a string-based log entry.

Django leverages the potential of logging by using the Python logging module by default, which provides different ways to create customized loggers through handlers, formats, and levels. The logging module is capable of:

• Multithreading execution
• Categorizing messages via different log levels
• Setting the destination of the logs
• Controlling what to include and what to emit

How to Add Logging in Django

To use logging in your Django project, follow these steps.

Configure the settings.py for various loggers, handlers, and filters:

import logging.config
import os
from django.utils.log import DEFAULT_LOGGING

# Disable Django's logging setup
LOGGING_CONFIG = None

LOGLEVEL = os.environ.get('LOGLEVEL', 'info').upper()

logging.config.dictConfig({
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'default': {
            # exact format is not important, this is the minimum information
            'format': '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
        },
        'django.server': DEFAULT_LOGGING['formatters']['django.server'],
    },
    'handlers': {
        # console logs to stderr
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'default',
        },
        'django.server': DEFAULT_LOGGING['handlers']['django.server'],
    },
    'loggers': {
        # default for all undefined Python modules
        '': {
            'level': 'WARNING',
            'handlers': ['console'],
        },
        # Our application code
        'app': {
            'level': LOGLEVEL,
            'handlers': ['console'],
            # Avoid double logging because of root logger
            'propagate': False,
        },
        # Default runserver request logging
        'django.server': DEFAULT_LOGGING['loggers']['django.server'],
    },
})

Restart the server. You’ll be able to see logs on your console or log files, depending on the configuration.

Best Practices for Logging in Django

Logging is vital to your Django application because it can save you time in crucial situations. Follow these best practices for implementation:

• Create loggers using the logging.getLogger() factory function, so that the logging library can manage the mapping of names to instances and maintain a hierarchy of logs. This way, you can use the logger’s name to access it in different parts of the application, and only a set number of loggers will be created at runtime.

• Specify proper log levels to lower the risks of deployment and ensure effective debugging. This helps prevent the flooding of log files with trivial information due to inappropriate settings.

• Format logs correctly so that the system can parse them. This is useful when manually reading the logs isn’t enough, such as for audits or alerts.

• Don’t log sensitive information like passwords, authorization tokens, personally identifiable information (PII), credit card numbers, or session identifiers that the user has chosen.

• Use fault-tolerant protocols while transferring logs to avoid packet loss. Secure log data by encrypting it and removing any sensitive information before transferring it.

• Create meaningful log messages so that you can easily tell what happened from the log file.

• Enhance log messages by including additional information.

• Don’t make log messages reliant on the content of prior messages, since the previous messages may not display if they’re logged at different levels.

• Ensure that logs are written asynchronously during log generation. Buffer or queue logs to prevent the program from stalling. Organize logs so that it’s simple to make changes as needed.

• Use a wrapper to shield the program from third-party tools. Use standard date and time formats, add timestamps, use log levels appropriately, and include a stack trace when reporting the error to make logs more human-readable. Include the thread’s name in a multithreaded program.

• Use filters or logging.LoggerAdapter to inject local contextual information, or use logging.setLogRecordFactory() to inject global contextual information in the log records. Don’t log too much, though, or it might become difficult to extract value.

• If you use FileHandler to write logs, the log file will expand over time and eventually take up all your storage space. In the production environment, utilize RotatingFileHandler instead of FileHandler to prevent this problem.

• When you have many different servers and log files, set up a central log system for all critical messages so you can quickly monitor it for problems.

Logging configuration in Django is simple, but it can get complicated when dealing with large applications. Aside from the Python logging module, you can also use popular logging tools such as NewRelic.

Final Thoughts

Logging can help you improve your application development and end-user experience. Because Django is a Python-based framework, Python’s logging system benefits you in several ways, and you can implement it fairly easily. Remember to follow the above best practices to simplify your setup and ensure better-quality results.