H A R S H H A A for ProDevOpsGuy Tech Community

Posted on Oct 26

DevOps Automation with Shell Scripting 🚀

#devops #automation #bash #beginners

Table of Contents

Introduction
Why Automate with Shell Scripting?
Basic Concepts of Shell Scripting
- 3.1 Essential Shell Commands
- 3.2 Variables and Data Types
- 3.3 Conditionals
- 3.4 Loops
Setting Up Your Environment for Automation
Shell Scripting for DevOps Use Cases
- 5.1 System Monitoring
- 5.2 Automating Deployments
- 5.3 Backup and Restore
- 5.4 Log Management
Advanced Shell Scripting Techniques
Best Practices in Shell Scripting for DevOps
Hands-On Projects for Practice
Conclusion

1. Introduction

In the world of DevOps, automation is key to managing complex systems, streamlining workflows, and accelerating development cycles. Shell scripting is a versatile and powerful tool that allows DevOps engineers to automate routine tasks, reducing manual effort and ensuring reliability across processes. In this guide, we will cover the essentials of DevOps automation with shell scripting, exploring various use cases, techniques, and best practices.

2. Why Automate with Shell Scripting?

Shell scripts are widely used for automation because:

They’re lightweight and quick to execute on almost any system with a shell (like Bash on Linux).
They can integrate with other automation tools in DevOps, like Jenkins, Ansible, and Kubernetes.
Shell scripting is often faster to implement for simple automation tasks compared to heavier programming languages.

3. Basic Concepts of Shell Scripting

Before diving into specific use cases, let’s review some basics of shell scripting that are essential for any DevOps automation project.

3.1 Essential Shell Commands

Some fundamental shell commands you should be familiar with include:

echo: Print text to the terminal.
cat, tail, head: Work with file content.
grep: Search for patterns in files.
awk, sed: For text processing.
curl, wget: Fetch data from the web.

Example:

echo "Hello, DevOps!"
cat /etc/passwd | grep "user_name"

3.2 Variables and Data Types

Shell scripting has basic variable support. Declare variables as follows:

NAME="DevOpsEngineer"
echo "Hello, $NAME"

Variables are created without explicit types, but you can use declare for specific types (like -i for integers).
Use export to make a variable accessible to subprocesses.

3.3 Conditionals

Use if, elif, and else to create conditional statements.

if [ $USER == "devops" ]; then
    echo "Welcome, DevOps Engineer!"
else
    echo "Access denied."
fi

== and != are used for string comparison.
-eq, -ne, -gt, -lt, etc., are used for integer comparisons.

3.4 Loops

Loops are essential for automating repetitive tasks.

for loop:

  for i in {1..5}; do
      echo "Number $i"
  done

while loop:

  count=1
  while [ $count -le 5 ]; do
      echo "Count $count"
      ((count++))
  done

4. Setting Up Your Environment for Automation

Ensure you have a suitable environment:

Shell: Typically, Bash (/bin/bash) on Linux.
Text Editor: Use Vim, Nano, or any preferred editor for writing scripts.
Permissions: Grant executable permissions to scripts using chmod +x script.sh.

5. Shell Scripting for DevOps Use Cases

Now, let’s go over specific DevOps tasks that can be automated using shell scripting.

5.1 System Monitoring

Monitoring system resources like CPU, memory, and disk usage can be automated to alert the DevOps team when resources exceed thresholds.

Example script for CPU usage monitoring:

#!/bin/bash
THRESHOLD=80
cpu_usage=$(top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | awk '{print 100 - $1}')
if (( $(echo "$cpu_usage > $THRESHOLD" |bc -l) )); then
    echo "CPU usage is above $THRESHOLD%. Current usage: $cpu_usage%" | mail -s "CPU Alert" you@example.com
fi

5.2 Automating Deployments

Shell scripts can simplify deployments by pulling the latest code, building, and deploying it to a server.

Example deployment script:

#!/bin/bash
echo "Starting deployment..."
git pull origin main
docker build -t myapp:latest .
docker stop myapp || true
docker rm myapp || true
docker run -d --name myapp -p 80:80 myapp:latest
echo "Deployment completed."

5.3 Backup and Restore

Regularly backing up data is crucial. Shell scripts can automate backup tasks and store them in secure locations.

Example for MySQL database backup:

#!/bin/bash
DATE=$(date +%F)
DB_BACKUP_PATH='/backup/dbbackup'
DB_USER='root'
DB_PASS='password'
DB_NAME='mydb'

mkdir -p ${DB_BACKUP_PATH}/${DATE}
mysqldump -u${DB_USER} -p${DB_PASS} ${DB_NAME} > ${DB_BACKUP_PATH}/${DATE}/${DB_NAME}.sql
echo "Backup completed for database ${DB_NAME}."

5.4 Log Management

Log files can grow large over time. Shell scripts can archive, rotate, or delete old logs to save space.

Example for archiving logs:

#!/bin/bash
LOG_DIR="/var/log/myapp/"
ARCHIVE_DIR="/var/log/myapp/archive/"
find $LOG_DIR -name "*.log" -mtime +30 -exec mv {} $ARCHIVE_DIR \;
gzip $ARCHIVE_DIR/*.log
echo "Log files older than 30 days archived and compressed."

6. Advanced Shell Scripting Techniques

Advanced shell scripting can make scripts more powerful and adaptable. Here are a few techniques:

Functions: Break scripts into reusable functions.

  function backup() {
      echo "Performing backup..."
      # Backup logic here
  }
  backup

Error Handling: Use set -e at the beginning to exit on errors or use traps to handle errors gracefully.
Parsing Arguments: Allow flexibility by parsing command-line arguments.

  while getopts u:p: flag
  do
      case "${flag}" in
          u) user=${OPTARG};;
          p) pass=${OPTARG};;
      esac
  done
  echo "User: $user"
  echo "Password: $pass"

7. Best Practices in Shell Scripting for DevOps

Following best practices will improve the readability, reliability, and performance of your scripts:

Use Comments: Comment your code for clarity.
Use Descriptive Variable Names: Improve readability.
Modularize with Functions: Keep code organized by using functions.
Error Handling: Handle potential errors with conditions or traps.
Limit Root Access: Run scripts with the least privilege necessary.

8. Hands-On Projects for Practice

Here are some hands-on projects to help you practice and improve your DevOps shell scripting skills:

Automated Server Provisioning: Write a script that sets up a web server (e.g., Apache or Nginx) on a new Linux instance.
CI/CD Pipeline Script: Write a shell script to automate steps of a CI/CD pipeline (e.g., cloning, building, and deploying).
User Access Management: Automate user creation, permission setting, and access logging.
Automated Health Check: Create a script that checks the health of critical services and sends alerts if services are down.

9. Conclusion

Shell scripting is an essential skill for any DevOps engineer. By mastering shell scripting, you can automate repetitive tasks, increase efficiency, and ensure the consistency and reliability of processes. Whether you're managing deployments, backups, or monitoring, shell scripting provides a quick and effective way to automate key DevOps tasks. Continue to practice with real-world projects to enhance your scripting skills and keep exploring advanced techniques to take your DevOps automation to the next level.