Oliver Bennet

Posted on Oct 19 • Originally published at graphpe.com

Automating Server Health Checks with Python, Bash, and PowerShell

#python #bash #automation #webdev

In today's fast-paced IT landscape, maintaining server uptime is crucial for system administrators and DevOps engineers. Automated server health checks offer a powerful solution, drastically reducing downtime risks through real-time monitoring of vital metrics such as CPU usage, memory consumption, and disk space. This post delves into automating these checks using three versatile scripting languages: Python, Bash, and PowerShell.

We'll provide step-by-step examples for creating these scripts and discuss the benefits of automating server health checks. By the end of this post, you'll understand why this automation is essential and how to implement it effectively in your infrastructure.

The Importance of Server Health Checks

Server health checks are an integral part of system monitoring. They allow you to keep a constant watch over your server's critical metrics, ensuring that you’re aware of any performance bottlenecks or potential issues before they escalate into full-blown problems. Failing to monitor servers can lead to serious consequences, including:

Unexpected Downtime: A server that runs out of memory or disk space without warning can suddenly go offline, impacting productivity, customer experience, and revenue.
Performance Degradation: Servers operating under high load may not crash, but their performance could degrade significantly, leading to sluggish applications, delayed response times, and unhappy users.
Data Loss: When storage resources are critically low, especially on database servers, you risk data corruption or loss.

With server health monitoring, you can catch these issues early, allowing your team to address them before they impact your services. Traditionally, these checks were performed manually, but with the complexity of modern infrastructures, automation has become necessary to handle the increasing number of servers and services.

Why Automate Server Health Checks?

Manually checking server performance metrics is both time-consuming and prone to human error. Here’s why automation is key:

1. Proactive Problem Detection

Automated scripts can run at regular intervals (every minute, hour, or day) and notify you before small issues become bigger problems. For instance, if a server's CPU consistently hits 90% usage, this could be an early sign of an application or process consuming more resources than expected. Early detection allows you to optimize your systems before they fail.

2. Consistency

Automated scripts ensure that checks are performed uniformly across all servers. This consistency helps avoid situations where a server is overlooked during manual monitoring, which is common in environments with multiple servers. Automation ensures that each server receives the same level of attention without any omissions.

3. Time Efficiency

Manually monitoring a fleet of servers can take hours, if not days, especially when multiple checks (CPU, memory, disk, network, etc.) are required. Automated scripts reduce this process to minutes or even seconds, allowing system administrators to focus on higher-priority tasks.

4. Customizable Alerts

Automation allows you to set custom alert triggers. For example, if a server’s disk space usage exceeds 80%, the script can send an email, push a notification, or even log the issue in a monitoring system like Prometheus or Grafana. This way, the right people are alerted instantly, preventing service disruption.

Defining Thresholds for Monitoring

Before diving into the scripts, it's important to understand the thresholds you will set for server monitoring. Thresholds are limits beyond which you consider a particular resource to be under heavy usage or at risk. Setting appropriate thresholds is crucial because overly sensitive thresholds could lead to false alerts, while overly lax thresholds may cause you to miss critical warnings.

CPU Threshold: A common threshold for CPU usage is around 80%. When CPU usage exceeds this percentage, it indicates that the server is handling heavy workloads, and you may need to optimize the processes or scale resources.
Memory Threshold: For memory usage, a threshold of 80% is typical as well. Consistently high memory usage could indicate memory leaks or applications that need more RAM.
Disk Space Threshold: A threshold of 80% is generally recommended for disk space. Low disk space can lead to failures in writing log files or database transactions, potentially causing data loss.

When choosing these thresholds, consider your specific environment and workload. For instance, CPU-heavy applications may regularly consume more than 80%, but that could be normal for your usage patterns. It’s essential to balance sensitivity with real-world requirements.

Automating Health Checks with Python

Python is an excellent language for system monitoring due to its versatility and robust libraries like psutil. Here’s how you can monitor CPU, memory, and disk usage and send email alerts if these metrics exceed the predefined thresholds.

How Python Enhances Server Monitoring

Cross-Platform: Python scripts can run on both Linux and Windows servers, making it versatile for mixed environments.
Rich Libraries: The psutil library allows for easy access to system performance metrics.
Integration: Python can easily integrate with external services like email, Slack, or monitoring tools such as Grafana for alerting.

You can use Python to extend monitoring to network interfaces, running processes, and even system temperatures, depending on your requirements.

import psutil
import smtplib
from email.mime.text import MIMEText

# Thresholds
CPU_THRESHOLD = 80  # in percent
MEMORY_THRESHOLD = 80  # in percent
DISK_THRESHOLD = 80  # in percent
DISK_PARTITION = '/'  # Root partition

def send_alert(subject, body):
    sender_email = "youremail@example.com"
    receiver_email = "admin@example.com"
    msg = MIMEText(body)
    msg['Subject'] = subject
    msg['From'] = sender_email
    msg['To'] = receiver_email

    with smtplib.SMTP('smtp.example.com', 587) as server:
        server.starttls()
        server.login(sender_email, "your_password")
        server.sendmail(sender_email, receiver_email, msg.as_string())

def check_cpu():
    cpu_usage = psutil.cpu_percent(interval=1)
    if cpu_usage > CPU_THRESHOLD:
        send_alert(f"High CPU Usage Alert: {cpu_usage}%", f"CPU usage is {cpu_usage}%")

def check_memory():
    memory = psutil.virtual_memory()
    memory_usage = memory.percent
    if memory_usage > MEMORY_THRESHOLD:
        send_alert(f"High Memory Usage Alert: {memory_usage}%", f"Memory usage is {memory_usage}%")

def check_disk():
    disk = psutil.disk_usage(DISK_PARTITION)
    disk_usage = disk.percent
    if disk_usage > DISK_THRESHOLD:
        send_alert(f"Low Disk Space Alert: {disk_usage}%", f"Disk usage is {disk_usage}% on {DISK_PARTITION}")

if __name__ == "__main__":
    check_cpu()
    check_memory()
    check_disk()

Bash for Server Health Monitoring in Linux

For Linux administrators, Bash is the go-to scripting language. It’s lightweight, fast, and comes pre-installed on most Unix-based systems. Automating health checks with Bash is straightforward using built-in Linux utilities like top, df, and free.

Why Use Bash for Health Checks?

Efficiency: Bash scripts execute faster because they don’t rely on external libraries.
No Dependencies: All commands used in a typical Bash monitoring script (top, df, etc.) are part of standard Linux distributions.
Perfect for Cron Jobs: Bash scripts are easy to schedule using cron, which allows for periodic execution of tasks.

Bash scripts are especially useful for basic system monitoring tasks where you want a fast, lightweight solution.

Example Bash Script:

#!/bin/bash

# Thresholds
CPU_THRESHOLD=80
MEMORY_THRESHOLD=80
DISK_THRESHOLD=80
PARTITION="/"

# Send alert email function
send_alert() {
    SUBJECT="$1"
    MESSAGE="$2"
    echo "$MESSAGE" | mail -s "$SUBJECT" admin@example.com
}

# CPU Check
cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2 + $4}')
if (( ${cpu_usage%.*} > CPU_THRESHOLD )); then
    send_alert "High CPU Usage" "Current CPU usage is $cpu_usage%"
fi

# Memory Check
memory_usage=$(free | awk '/Mem/{printf("%.0f"), $3/$2 * 100.0}')
if (( memory_usage > MEMORY_THRESHOLD )); then
    send_alert "High Memory Usage" "Current memory usage is $memory_usage%"
fi

# Disk Check
disk_usage=$(df -h | grep $PARTITION | awk '{print $5}' | sed 's/%//g')
if (( disk_usage > DISK_THRESHOLD )); then
    send_alert "Low Disk Space" "Current disk usage is $disk_usage% on $PARTITION"
fi

PowerShell for Windows Server Monitoring

For Windows environments, PowerShell is an ideal tool for automating server health checks. It offers deep integration with Windows APIs and systems, allowing for comprehensive monitoring without the need for third-party tools.

PowerShell’s Strength in Monitoring

Windows Native: PowerShell is built into Windows, making it the perfect choice for server management in Windows environments.
Advanced Scripting Capabilities: PowerShell can query the Windows Management Instrumentation (WMI) to retrieve detailed system metrics.
Remote Management: PowerShell scripts can be run remotely, making it easier to manage multiple servers across a network.

PowerShell also offers integration with Windows Task Scheduler, so you can run scripts automatically at predefined intervals.

Example PowerShell Script:

# Thresholds
$CPU_THRESHOLD = 80
$MEMORY_THRESHOLD = 80
$DISK_THRESHOLD = 80
$DISK_DRIVE = "C:"

# Send alert email function
function Send-Alert {
    param([string]$Subject, [string]$Body)
    $smtpServer = "smtp.example.com"
    $smtpFrom = "admin@example.com"
    $smtpTo = "admin@example.com"
    $message = New-Object system.net.mail.mailmessage
    $message.From = $smtpFrom
    $message.To.Add($smtpTo)
    $message.Subject = $Subject
    $message.Body = $Body
    $smtp = New-Object Net.Mail.SmtpClient($smtpServer)
    $smtp.Send($message)
}

# CPU Check
$cpu = Get-WmiObject win32_processor | Measure-Object -property LoadPercentage -Average | Select-Object Average
if ($cpu.Average -gt $CPU_THRESHOLD) {
    Send-Alert "High CPU Usage" "Current CPU usage is $($cpu.Average)%"
}

# Memory Check
$memory = Get-WmiObject win32_operatingsystem
$memoryUsage = (($memory.TotalVisibleMemorySize - $memory.FreePhysicalMemory) / $memory.TotalVisibleMemorySize) * 100
if ($memoryUsage -gt $MEMORY_THRESHOLD) {
    Send-Alert "High Memory Usage" "Current memory usage is $([math]::Round($memoryUsage,2))%"
}

# Disk Check
$disk = Get-WmiObject win32_logicaldisk -Filter "DeviceID='$DISK_DRIVE'"
if ($disk.FreeSpace / $disk.Size * 100 -lt $DISK_THRESHOLD) {
    Send-Alert "Low Disk Space" "Current disk usage on drive $DISK_DRIVE is $([math]::Round((($disk.Size - $disk.FreeSpace) / $disk.Size) * 100, 2))%"
}

Customizing Health Check Scripts for Different Environments

Whether you're using Python, Bash, or PowerShell, you can customize these scripts to meet specific needs within your environment. Here are some examples:

Cloud Environments (AWS, Azure, GCP):

Add cloud-specific monitoring, like checking for CPU credits on AWS EC2 instances or tracking Azure disk performance metrics. Python’s boto3 library or PowerShell’s Az module can help integrate cloud-specific health checks.
Database Servers:

Monitor not only system resources but also database-specific metrics. For example, you can check PostgreSQL or MySQL disk usage, running queries, and connection pool utilization.
Containers and Microservices:

For environments running Docker or Kubernetes, you can extend the monitoring scripts to check container CPU and memory usage or even orchestrator-specific metrics like pod restarts.

Advanced Alerting and Integration with Monitoring Tools

While these scripts send basic email alerts, you can easily extend them to integrate with modern monitoring solutions like Prometheus, Grafana, or Zabbix. For example:

Slack Alerts: Instead of email, send alerts to a Slack channel for immediate visibility by the team.
SMS Notifications: Use services like Twilio to send SMS notifications if a server goes down.
Log Management: Ship alerts to a centralized log management system like ELK Stack (Elasticsearch, Logstash, Kibana) for further analysis.

By integrating your scripts with these tools, you can build a robust monitoring system that scales with your infrastructure and ensures you are always in the loop.

Conclusion

Automating server health checks is essential for maintaining a reliable, scalable IT infrastructure. Whether you're managing a handful of servers or a large-scale cloud environment, Python, Bash, and PowerShell provide flexible, powerful options for automating these critical tasks. By implementing these scripts, you can proactively monitor your systems, avoid downtime, and maintain high performance.

In next week’s Scripting Saturdays, we’ll dive into automating backup and restore processes using Python, Bash, and PowerShell—so stay tuned!

What do you think about these health check scripts? Have you implemented server health monitoring in your environment? Let us know your thoughts in the comments below, or reach out if you have questions about customizing these scripts for your needs.

🔗 Related Links:
▶️ Website: https://graphpe.com
▶️ Subscribe: Subscribe on YouTube
▶️ Follow me on X for updates: https://x.com/obennetgpe
▶️ Check out my articles on dev.to: https://dev.to/oliverbennet
☕ Buy me a Coffee Here: https://buymeacoffee.com/graphpe

👨‍💻 Related Videos:
⏩ How to Install Ubuntu 24.04 on Proxmox
⏩ How to Install Grafana on Ubuntu 24

Oliver | GraphPe | Tutorials

DEV Community