Day 14: Advanced Logging and Monitoring with Prometheus, Grafana, and Alertmanager (Termux Edition!)

Introduction

Today, we will look deep into the world of advanced logging and monitoring, but with a twist—we're doing it all on our phones! This is Day 14 of my Devops project, and we're going mobile with Termux( I don't have access to my windows machine because I will be on transit all day but I thought it would be fun to see what I can do using termux) to build a powerful monitoring system using Prometheus, Grafana, and Alertmanager. This is a great way to learn and practice these tools, even when you're on the go.

Objectives

• Create a Node.js application that generates metrics and logs.
• Dockerize the application for easy deployment.
• Configure Prometheus to collect metrics and trigger alerts.
• Set up Alertmanager to send email to notify you.
• Use Grafana to visualize metrics and create dashboards.
• Organize the project for easy sharing on GitHub.

A More detailed Guide

1. Setting up the Node.js Application (Termux)

• Create the Project Directory:
  

   mkdir Day14-monitoring-app
   cd Day14-monitoring-app
  

• Create the app Directory:
  

   mkdir app
   cd app
  

• Create the index.js File:
  

   touch index.js
  

• Open index.js in Termux's Nano editor:
  

   nano index.js
  

• Paste the following code into index.js:
  

   // Import necessary modules
   const express = require('express');
   const app = express();
   const port = 4000;
   const client = require('prom-client');
   const morgan = require('morgan');
   const collectDefaultMetrics = client.collectDefaultMetrics;
  
   // Collect default metrics (CPU, memory, etc.)
   collectDefaultMetrics();
  
   // Middleware for logging requests
   app.use(morgan('combined')); // Logs HTTP requests
  
   // Define routes for our application
   app.get('/', (req, res) => res.send('Hello World!'));
   app.get('/api/data', (req, res) => res.json({ data: 'some data' }));
  
   // Endpoint to expose Prometheus metrics
   app.get('/metrics', (req, res) => {
     res.set('Content-Type', client.register.contentType);
     res.end(client.register.metrics());
   });
  
   // Start the server
   app.listen(port, () => console.log(`App listening at http://localhost:${port}`));
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

• Create the package.json File:
  

   touch package.json
  

• Open package.json in Nano:
  

   nano package.json
  

• Paste the following code into package.json:
  

   {
     "name": "advanced-monitoring-app",
     "version": "1.0.0",
     "description": "A Node.js application with advanced monitoring",
     "main": "index.js",
     "scripts": {
       "start": "node index.js"
     },
     "author": "Your Name",
     "license": "MIT",
     "dependencies": {
       "express": "^4.18.2",
       "morgan": "^1.10.0",
       "prom-client": "^16.0.0"
     }
   }
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

• Install Dependencies:
  

   npm install
  

2. Dockerizing the Application (Termux)

• Create the Dockerfile:
  

   touch Dockerfile
  

• Open Dockerfile in Nano:
  

   nano Dockerfile
  

• Paste the following code into Dockerfile:
  

   # Use the Node.js 14 image as our base
   FROM node:14
  
   # Set the working directory inside the container
   WORKDIR /app
  
   # Copy the package.json file
   COPY package*.json ./
  
   # Install dependencies using npm
   RUN npm install
  
   # Copy the rest of the application code
   COPY . .
  
   # Expose port 4000 for our application to listen on
   EXPOSE 4000
  
   # Run the Node.js application when the container starts
   CMD ["node", "index.js"] 
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

• Build the Docker Image:
  

   docker build -t your_dockerhub_username/advanced-monitoring-app:latest .
  

• Log in to Docker Hub:
  

   docker login
  

• Push the Image to Docker Hub:
  

   docker push your_dockerhub_username/advanced-monitoring-app:latest
  

3. Configuring Prometheus (Termux)

• Create the prometheus.yml File:
  

   touch prometheus.yml
  

• Open prometheus.yml in Nano:
  

   nano prometheus.yml
  

• Paste the following code into prometheus.yml:
  

   global:
     scrape_interval: 15s # Scrape metrics every 15 seconds
  
   scrape_configs:
     - job_name: 'node-app' # Name of the job for scraping our app
       static_configs:
         - targets: ['host.docker.internal:4000'] # Target to scrape (our app)
   rule_files:
     - 'alert.rules.yml'
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

4. Configuring Alertmanager (Termux)

• Create the alertmanager.yml File:
  

   touch alertmanager.yml
  

• Open alertmanager.yml in Nano:
  

   nano alertmanager.yml
  

• Paste the following code into alertmanager.yml:
  

   global:
     resolve_timeout: 5m # Alerts are considered resolved after 5 minutes of no new alerts
  
   route:
     group_by: ['job'] # Group alerts by job name
     group_wait: 30s # Wait 30 seconds to group alerts
     group_interval: 5m # Group alerts every 5 minutes
     repeat_interval: 12h # Repeat alerts every 12 hours
     receiver: 'email' # Send alerts to the email receiver
  
   receivers:
     - name: 'email'
       email_configs:
         - to: 'your-email@example.com' # Your email address
           from: 'alertmanager@example.com' # Sender email
           smarthost: 'smtp.example.com:587' # SMTP server (replace with your settings)
           auth_username: 'alertmanager' # SMTP username
           auth_password: 'password' # SMTP password
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

5. Docker Compose Setup (Termux)

• Create the docker-compose.yml File:
  

   touch docker-compose.yml
  

• Open docker-compose.yml in Nano:
  

   nano docker-compose.yml
  

• Paste the following code into docker-compose.yml:
  

   version: '3'
  
   services:
     prometheus:
       image: prom/prometheus
       volumes:
         - ./prometheus.yml:/etc/prometheus/prometheus.yml # Mount the prometheus.yml file
       ports:
         - "9090:9090" # Expose port 9090 for Prometheus
  
     grafana:
       image: grafana/grafana
       ports:
         - "3000:3000" # Expose port 3000 for Grafana
  
     alertmanager:
       image: prom/alertmanager
       volumes:
         - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml # Mount the alertmanager.yml file
       ports:
         - "9093:9093" # Expose port 9093 for Alertmanager
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

6. Adding Custom Alerts (Termux)

• Create the alert.rules.yml File:
  

   touch alert.rules.yml
  

• Open alert.rules.yml in Nano:
  

   nano alert.rules.yml
  

• Paste the following code into alert.rules.yml:
  

   groups:
     - name: example
       rules:
         - alert: HighRequestLatency
           expr: http_request_duration_seconds_mean{job="node-app"} > 0.5 # Check if average request duration is over 0.5 seconds
           for: 2m # Trigger alert only if the condition is true for 2 minutes
           labels:
             severity: "page" # Add a label to the alert
           annotations:
             summary: "High request latency" # Alert summary
             description: "The request latency is above 0.5s for more than 2 minutes." # Detailed description
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

7. Deploying the Stack (Termux)

   # Start the Docker Compose stack in detached mode
   docker-compose up -d

8. Configuring Grafana Dashboards (Termux)

• Access Grafana: Open your web browser and go to http://localhost:3000.
• Add Prometheus as a data source: In Grafana, add Prometheus as a data source.
• Create custom dashboards: Create dashboards to visualize your metrics.
• Set up alerts in Grafana: Set up alerts in Grafana to trigger notifications via Alertmanager.

9. GitHub Actions Workflow (Termux)

• Create the .github/workflows Directory:
  

   mkdir .github/workflows
   cd .github/workflows
  

• Create the main.yml File:
  

   touch main.yml
  

• Open main.yml in Nano:
  

   nano main.yml
  

• Paste the following code into main.yml:
  

   name: Docker Build & Deploy
  
   on:
     push:
       branches:
         - main
  
   jobs:
     build-and-deploy:
       runs-on: ubuntu-latest
       steps:
         - name: Checkout Repository
           uses: actions/checkout@v2
  
         - name: Login to Docker Hub
           uses: docker/login-action@v2
           with:
             username: ${{ secrets.DOCKER_USERNAME }}
             password: ${{ secrets.DOCKER_PASSWORD }}
  
         - name: Build and Push Docker Image
           run: |
             docker build -t ${{ secrets.DOCKER_USERNAME }}/advanced-monitoring-app:latest .
             docker push ${{ secrets.DOCKER_USERNAME }}/advanced-monitoring-app:latest
  
         - name: Deploy Docker Container
           run: |
             docker-compose up -d
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

10. Setting Up Your GitHub Repository

• Create a new repository on GitHub.

• Initialize a Git repository in your project directory:
  

   git init
  

• Add all files to the repository:
  

   git add .
  

• Commit your changes:
  

   git commit -m "Initial commit"
  

• Add the GitHub remote:
  

   git remote add origin <your_github_repo_url>
  

• Push your changes to GitHub:
  

   git push origin main
  

Remember:

• Replace your_dockerhub_username with your actual Docker Hub username.
• Replace your-email@example.com with your email address.
• Replace smtp.example.com:587 with your SMTP server address and port.
• Set the DOCKER_USERNAME and DOCKER_PASSWORD secrets in your GitHub repository settings.

Conclusion

With this setup , you can indeed monitor your application and receive alerts directly on your phone using Termux. Here's how it works:

1. Data Collection: The Node.js application you've created acts as a source of data. It collects metrics about HTTP requests, system resources (CPU, memory), and network connections.

2. Prometheus: Prometheus is the data collector and storage engine. It regularly scrapes metrics from your Node.js application and stores them.

3. Alertmanager: Alertmanager is the notification system. You've configured it to send email alerts when certain conditions are met. For example, if your Node.js application experiences high request latency or if your system's CPU load exceeds a threshold, Alertmanager will send you an email notification.

4. Grafana: Grafana is the visualization tool. It allows you to create dashboards to view and analyze the collected metrics. You can create graphs, charts, and other visualizations to gain insights into your application's performance.

5. Termux: Termux is your mobile terminal environment. You can use it to run all the necessary commands to build, deploy, and manage your monitoring stack. You can also use Termux to access Grafana's web interface on your phone's browser to view your dashboards and monitor your application.

How to Receive Alerts on Your Phone:

• Email: The most straightforward way to receive alerts is through email. You've already configured Alertmanager to send emails to your specified address.
• Push Notifications: You can explore integrations with push notification services like Pushover or Prowl. These services allow you to receive alerts directly on your phone as push notifications.

Key Points:

• Termux is your mobile command center: Which can start and stop the Docker Compose stack, view logs, and manage your monitoring system.
• Grafana is your visual dashboard: It gives you a clear and intuitive way to see your application's performance.
• Alertmanager is your early warning system: It notifies you of potential problems so you can address them quickly.

-To receive emails on your phone, you'll need to ensure that your email client is configured .

• If you want to use push notifications, you'll need to set up an account with a push notification service and integrate it with Alertmanager.

With this setup, you have a powerful and flexible monitoring system at your fingertips, even when you're on the go!