Advanced Open edX Monitoring with AppSignal for Python

In the first part of this series, we explored how AppSignal can significantly enhance the robustness of Open edX platforms. We saw the challenges that Open edX faces as it scales and how AppSignal's features — including real-time performance monitoring and automated error tracking — provide essential tools for DevOps teams. Our walkthrough covered the initial setup and integration of AppSignal with Open edX, highlighting the immediate benefits of this powerful observability framework.

In this second post, we'll dive deeper into the advanced monitoring capabilities that AppSignal offers. This includes streaming logs from Open edX to AppSignal, monitoring background workers with Celery, and tracking Redis queries. We will demonstrate how these features can be leveraged to address specific operational challenges, ensuring that our learning platform remains fail-safe under varying circumstances.

By the end of this article, you will know how to utilize AppSignal to its full potential in maintaining and improving the performance and reliability of your Open edX platform.

Streaming Logs to AppSignal

One of AppSignal's strongest features is centralized log management.

Commonly at Open edX, the support team reports an issue with the site, and an engineer can SSH into the server right away to check for Nginx, Mongo, MySQL, and Open edX Application logs.

A centralized storage place that houses logs without the need for you to SSH into the server is a really powerful feature. We can also set up notifications based on an issue's severity.

Now let's see how we can stream our logs from Open edX to AppSignal.

Create a Source

Under the Logging section, click on Manage sources and create a new source, with HTTP as the platform and JSON as the format. After creating the source, AppSignal provides an endpoint and API KEY that we can POST our logs to.

To have more control over log transmission, we can write a simple Python script that reads logs from our local Open edX, pre-processes them, and moves the important ones to AppSignal. For example, I wrote the following script to move only ERROR logs to AppSignal (skipping INFO and WARNING logs):

import requests
import json
from datetime import datetime
import logging

# Setup logging configuration
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# File to keep track of the last processed line
log_pointer_file = '/root/.local/share/tutor/data/lms/logs/processed.log'
log_file = '/root/.local/share/tutor/data/lms/logs/all.log'

# APpSignal API KEY
api_key = "MY-API-KEY"  # Replace with your actual API key
# URL to post the logs
url = f'https://appsignal-endpoint.net/logs?api_key={api_key}'

def read_last_processed():
    try:
        with open(log_pointer_file, 'r') as file:
            content = file.read().strip()
            last_processed = int(content) if content else 0
            logging.info(f"Last processed line number read: {last_processed}")
            return last_processed
    except (FileNotFoundError, ValueError) as e:
        logging.error(f"Could not read from log pointer file: {e}")
        return 0

def update_last_processed(line_number):
    try:
        with open(log_pointer_file, 'w') as file:
            file.write(str(line_number))
            logging.info(f"Updated last processed to line number: {line_number}")
    except Exception as e:
        logging.error(f"Could not update log pointer file: {e}")

def parse_log_line(line):
    if 'ERROR' in line:
        parts = line.split('ERROR', 1)
        timestamp = parts[0].strip()
        message_parts = parts[1].strip().split(' - ', 1)
        message = message_parts[1] if len(message_parts) > 1 else ''
        attributes_part = message_parts[0].strip('[]').split('] [')
        # Flatten attributes into a dictionary with string keys and values
        attributes = {}
        for attr in attributes_part:
            key_value = attr.split(None, 1)
            if len(key_value) == 2:
                key, value = key_value
                key = key.rstrip(']:').replace(' ', '_').replace('.', '_')  # Replace spaces and dots in keys
                if len(key) <= 50:
                    attributes[key] = value
        # Format the timestamp
        formatted_timestamp = datetime.strptime(timestamp, '%Y-%m-%d %H:%M:%S,%f').isoformat()[:-3] + 'Z'
        # Add the message and attributes to the log structure
        json_data = {
            "timestamp": formatted_timestamp,
            "group": "openedx",
            "severity": "error",
            "hostname": "tutor",
            "message": message,
        }
        json_data.update(attributes)  # Add the attributes directly to the json_data dictionary
        return json_data

def post_logs(json_data):
    headers = {'Content-Type': 'application/json'}
    response = requests.post(url, json=json_data, headers=headers)
    logging.info(f"Posted log to server; HTTP status code: {response.status_code}, Response: {response.content}")
    return response.status_code

def process_logs():
    last_processed = read_last_processed()
    with open(log_file, 'r') as file:
        for i, line in enumerate(file, 1):
            if i > last_processed:
                json_data = parse_log_line(line)
                if json_data:
                    response_code = post_logs(json_data)
                    if response_code == 200:
                        update_last_processed(i)
                    else:
                        logging.warning(f"Failed to post log, HTTP status code: {response_code}")

if __name__ == '__main__':
    logging.info("Starting log processing script.")
    process_logs()
    logging.info("Finished log processing.")

Here's how the script works:

1. Log File Management: Tutor saves all of the logs in the /root/.local/share/tutor/data/lms/logs/all.log file. This file contains MySQL, LMS, CMS, Caddy, Celery, and other services. The script uses a pointer /root/.local/share/tutor/data/lms/logs/processed.log file that tracks the last processed line. This ensures that each log is processed only once.
2. Error Filtering: As mentioned, we only send ERROR logs to AppSignal.
3. Data Parsing and Formatting: Each error log is parsed to extract key pieces of information, such as the timestamp and error message. The script formats this data into a JSON structure suitable for transmission.
4. Log Transmission: The formatted log data is sent to AppSignal using an HTTP POST request.

Important: Please make sure you don't send any personally identifiable information to the endpoint.

Now run this script and it should move ERROR logs to AppSignal:

You can also create a new trigger to notify you as soon as a specific event like ERROR happens:

Monitor Celery and Redis using AppSignal

Celery (a distributed task queue) is a vital component of Open edX, responsible for managing background tasks such as grading, certificate generation, and bulk email dispatch. Redis often acts as the broker for Celery, managing task queues. Both systems are essential for asynchronous processing and can become bottlenecks during periods of high usage. Monitoring these services with AppSignal provides valuable insights into task execution and queue health, helping you preemptively address potential issues. Let's see how we can monitor Celery and Redis.

First, install the necessary packages. Add the following to the OPENEDX_EXTRA_PIP_REQUIREMENTS variable in the .local/share/tutor/config.yml file:

- opentelemetry-instrumentation-celery==0.45b0
- opentelemetry-instrumentation-redis==0.45b0

It should look like the following:

OPENEDX_EXTRA_PIP_REQUIREMENTS:
  - appsignal==1.3.0
  - opentelemetry-instrumentation-django==0.45b0
  - opentelemetry-instrumentation-celery==0.45b0
  - opentelemetry-instrumentation-redis==0.45b0

As you can see, we are installing opentelemetry packages for Celery and Redis.

Now, we can instrument Celery with worker_process_init to report its metrics to AppSignal.

Heading back to our dashboard in AppSignal, we should see Celery and Redis reports in the Performance section, with background as the namespace.

For Redis queries, you can click on Slow queries:

Practical Monitoring: Enhancing Open edX with AppSignal

In this section, we'll revisit the initial issues outlined in part one of this series and apply practical AppSignal monitoring solutions to ensure our Open edX platform stays robust and reliable. Here’s a breakdown.

Site Performance Improvement

Let's begin by assessing overall site performance. In the Performance section, under the Issue list, we can see key metrics for all visited URLs:

• Response Time: Directly reflects user experience by measuring the time taken to process and respond to requests. Factors influencing this include database queries and middleware operations.
• Throughput: Indicates the number of requests handled within a given timeframe.
• Mean Response Time: Provides an average response time across all requests to a specific endpoint. Any mean response time over 1 second is a potential concern and highlights areas that need optimization.
• 90th Percentile Response Time: For example, a 90th percentile response time of 7 ms for GET store/ suggests that 90% of requests complete in 7 ms or less.

Now let's order all the actions based on the mean. Any item higher than 1 second should be considered a red flag:

As we see, Celery tasks to rescore and reset student attempts, LMS requests to show course content, and some APIs are taking more than 1 second. Also, we should note that this is only for one active user. If we have more concurrent users, this response time will go up. Our first solution is to add more resources to the server (CPU and memory) and do another performance test.

After identifying actions with mean response times exceeding 1 second, consider performance optimization strategies such as:

• Minimizing JavaScript execution
• Using CDNs for static content
• Implementing caching techniques.

Server Resource Monitoring

We talked about anomaly detection and host monitoring in the previous article. Let's add triggers for the following items:

• CPU usage
• Disk usage
• Memory usage
• Network traffic
• Error rate

Custom Metrics

Two really important metrics for our platform are our number of active users and enrollments. Let's see how we can measure these metrics using AppSignal.

First, add increment_counter to common/djangoapps/student/views/management.py and openedx/core/djangoapps/user_authn/views/login.py to track and increment the number of logins and enrollments when there is a new event.

Now let's log in to Open edX and enroll in a course. Next, let's head to our dashboard in AppSignal. Click on Add dashboard, then Create dashboard, and give it a name and description.

Click on Add graph, enter Active Users as the title, select Add Metric and use login_count:

Your dashboard should look like the following:

You can follow the same steps to add a graph for enrollments using an enrollment_count metric.

Ensuring Consistent Styling

To make sure our site's styling stays consistent, let's add a new uptime check for static/tailwind/css/lms-main-v1.css and get notified when a URL is broken:

Email Delivery and Error Handling

In the Error section of the dashboard, we can view all errors, set up notifications for them, and work on fixes as soon as possible to prevent users from being negatively impacted.

Background Job Efficiency for Grading

In the Monitor Celery and Redis section of this article, we saw how to instrument Celery and Redis using AppSignal. Let's follow the same steps to enable AppSignal so we can see graded tasks. In the lms/djangoapps/grades/tasks.py file, add the following lines:

We should now see a couple of items to grade under Performance -> Issue list.

As you can see, recalculate_subsection_grade_v3 (our main grading Celery task) takes 212 milliseconds. For regrading, lms.djangoapps.instructor_task.tasks.reset_problem_attempts and lms.djangoapps.instructor_task.tasks.rescore_problem take 1.77 seconds.

Wrapping Up

In this two-part series, we integrated AppSignal with Open edX to fortify its monitoring capabilities. We started with the basics — setting up and understanding the fundamental offerings of AppSignal, including error tracking and performance monitoring.

In this article, we tackled how to efficiently stream logs from various Open edX services to AppSignal, ensuring all relevant information was centralized and readily accessible. We also monitored crucial asynchronous tasks handled by Celery and Redis.

Finally, we addressed some real-world challenges, such as slow site responses, resource bottlenecks during high enrollment periods, and unexpected issues like broken styling.

By now, you should have a comprehensive understanding of how to leverage AppSignal to not just monitor, but also significantly improve, the performance and reliability of your Open edX platform.

If you have any questions about Open edX or need further assistance, feel free to visit cubite.io or reach out to me directly at amir@cubite.io.

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!