DEV Community

Cover image for 5 log parsing commands
cuongnp
cuongnp

Posted on

5 log parsing commands

#linux #beginners #development #programming

Have you ever tried to find something in the server log file? While downloading and opening the file in an editor might seem straightforward, it's often time-consuming and unproductive. Instead, using command-line tools can be more efficient and effective. Here are some common commands you should try.

The practice file today is system.log

2024-06-12 13:39:30 [INFO] Server started on port 8080
2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:41:05 [INFO] User 'john_doe' logged in
2024-06-12 13:42:16 [WARNING] Disk space low on /dev/sda1
2024-06-12 13:43:27 [INFO] Scheduled job 'backup' started
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:45:49 [INFO] User 'jane_smith' logged out
2024-06-12 13:46:50 [INFO] Server shutdown initiated
2024-06-12 13:47:51 [INFO] Server stopped
2024-06-12 13:48:52 [INFO] Server started on port 8080
2024-06-12 13:49:53 [INFO] User 'john_doe' logged in
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
2024-06-12 14:00:04 [WARNING] Disk space low on /dev/sda1
2024-06-12 14:01:05 [INFO] Scheduled job 'backup' started
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:03:07 [INFO] User 'jane_smith' logged out
2024-06-12 14:04:08 [INFO] Server shutdown initiated
2024-06-12 14:05:09 [INFO] Server stopped
2024-06-12 14:06:10 [INFO] Server started on port 8080
2024-06-12 14:07:11 [INFO] User 'john_doe' logged in
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:09:13 [WARNING] High memory usage detected
2024-06-12 14:10:14 [INFO] Scheduled job 'cleanup' started
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied
2024-06-12 14:12:16 [INFO] User 'john_doe' logged out
2024-06-12 14:13:17 [INFO] Server shutdown initiated
2024-06-12 14:14:18 [INFO] Server stopped
Enter fullscreen mode Exit fullscreen mode

1. Display the Contents of the Log File

cat Command

  • Purpose: used to display the content of files.
  • Usage: cat filename
  • Example: cat server.log 
$ cat system.log
2024-06-12 13:39:30 [INFO] Server started on port 8080
2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:41:05 [INFO] User 'john_doe' logged in
2024-06-12 13:42:16 [WARNING] Disk space low on /dev/sda1
2024-06-12 13:43:27 [INFO] Scheduled job 'backup' started
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:45:49 [INFO] User 'jane_smith' logged out
2024-06-12 13:46:50 [INFO] Server shutdown initiated
2024-06-12 13:47:51 [INFO] Server stopped
2024-06-12 13:48:52 [INFO] Server started on port 8080
2024-06-12 13:49:53 [INFO] User 'john_doe' logged in
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
2024-06-12 14:00:04 [WARNING] Disk space low on /dev/sda1
2024-06-12 14:01:05 [INFO] Scheduled job 'backup' started
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:03:07 [INFO] User 'jane_smith' logged out
2024-06-12 14:04:08 [INFO] Server shutdown initiated
2024-06-12 14:05:09 [INFO] Server stopped
2024-06-12 14:06:10 [INFO] Server started on port 8080
2024-06-12 14:07:11 [INFO] User 'john_doe' logged in
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:09:13 [WARNING] High memory usage detected
2024-06-12 14:10:14 [INFO] Scheduled job 'cleanup' started
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied
2024-06-12 14:12:16 [INFO] User 'john_doe' logged out
2024-06-12 14:13:17 [INFO] Server shutdown initiated
2024-06-12 14:14:18 [INFO] Server stopped
Enter fullscreen mode Exit fullscreen mode

2. Search for lines

grep

  • Purpose: powerful command for searching text using patterns, and filtering log entries based on specific criteria.
  • Usage: cat filename | grep “filter-condition” or grep condition filename
  • Example: grep "ERROR" server.log 
$ grep "ERROR" system.log

2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied
Enter fullscreen mode Exit fullscreen mode

3. Display Lines with Customize Condition

awk

  • Purpose: Introduce awk as a powerful text processing tool, ideal for manipulating data and generating reports.
  • Usage: awk condition filename
  • Example 1: Display lines with timestamps between 13:50:00 and 14:00:00: 
$ awk '/13:5[0-9]:[0-9][0-9]/ || /14:00:00/' system.log

2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
Enter fullscreen mode Exit fullscreen mode
  • Example 2: Extract and print the date and time of each entry 
$ awk '{print $1, $2}' system.log

2024-06-12 13:39:30
2024-06-12 13:40:12
2024-06-12 13:41:05
2024-06-12 13:42:16
2024-06-12 13:43:27
2024-06-12 13:44:38
2024-06-12 13:45:49
2024-06-12 13:46:50
2024-06-12 13:47:51
2024-06-12 13:48:52
2024-06-12 13:49:53
2024-06-12 13:50:54
2024-06-12 13:51:55
2024-06-12 13:52:56
2024-06-12 13:53:57
2024-06-12 13:54:58
2024-06-12 13:55:59
2024-06-12 13:56:00
2024-06-12 13:57:01
2024-06-12 13:58:02
2024-06-12 13:59:03
2024-06-12 14:00:04
2024-06-12 14:01:05
2024-06-12 14:02:06
2024-06-12 14:03:07
2024-06-12 14:04:08
2024-06-12 14:05:09
2024-06-12 14:06:10
2024-06-12 14:07:11
2024-06-12 14:08:12
2024-06-12 14:09:13
2024-06-12 14:10:14
2024-06-12 14:11:15
2024-06-12 14:12:16
2024-06-12 14:13:17
2024-06-12 14:14:18
Enter fullscreen mode Exit fullscreen mode

4. Sort Log Entries

sort

  • Purpose: Sort lines in text files.
  • Usage: Sort log entries by date, time, or any other field.
  • Example: cat system.log | awk '{print $1, $2, $3}' | sort 
$ cat system.log | awk '{print $1, $2, $3}' | sort
2024-06-12 13:39:30 [INFO]
2024-06-12 13:40:12 [ERROR]
2024-06-12 13:41:05 [INFO]
2024-06-12 13:42:16 [WARNING]
2024-06-12 13:43:27 [INFO]
2024-06-12 13:44:38 [ERROR]
2024-06-12 13:45:49 [INFO]
2024-06-12 13:46:50 [INFO]
2024-06-12 13:47:51 [INFO]
2024-06-12 13:48:52 [INFO]
2024-06-12 13:49:53 [INFO]
2024-06-12 13:50:54 [ERROR]
2024-06-12 13:51:55 [WARNING]
2024-06-12 13:52:56 [INFO]
2024-06-12 13:53:57 [ERROR]
2024-06-12 13:54:58 [INFO]
2024-06-12 13:55:59 [INFO]
2024-06-12 13:56:00 [INFO]
2024-06-12 13:57:01 [INFO]
2024-06-12 13:58:02 [ERROR]
2024-06-12 13:59:03 [INFO]
2024-06-12 14:00:04 [WARNING]
2024-06-12 14:01:05 [INFO]
2024-06-12 14:02:06 [ERROR]
2024-06-12 14:03:07 [INFO]
2024-06-12 14:04:08 [INFO]
2024-06-12 14:05:09 [INFO]
2024-06-12 14:06:10 [INFO]
2024-06-12 14:07:11 [INFO]
2024-06-12 14:08:12 [ERROR]
2024-06-12 14:09:13 [WARNING]
2024-06-12 14:10:14 [INFO]
2024-06-12 14:11:15 [ERROR]
2024-06-12 14:12:16 [INFO]
2024-06-12 14:13:17 [INFO]
2024-06-12 14:14:18 [INFO]
Enter fullscreen mode Exit fullscreen mode

5. Unique the display result

uniq

  • Purpose: Describe how uniq removes or counts duplicate lines.
  • Usage: cat filename | uniq -c
  • Example: cat server.log | grep "ERROR" | awk '{print $4}' | sort | uniq -c 
cat system.log | grep "ERROR" | awk '{print $4}' | sort | uniq -c
   2 Cleanup
   2 Could
   4 Failed
Enter fullscreen mode Exit fullscreen mode

Final thought

Analyzing information files is crucial for system administration, troubleshooting, and monitoring. Using a combination of command-line tools like cat, grep, awk, sort, and uniq, you can effectively manage and extract valuable insights from your log files.

Top comments (0)

Read next

nicooprat profile image

Using BroadcastChannel API with Vue to sync a ref across multiple tabs

Nico Prat -

rafaeljcamara profile image

Downstream Resiliency: The Timeout, Retry, and Circuit-Breaker Patterns

Rafael Câmara -

pvsdev profile image

How can one code line crash application? Looking for issues and vulnerabilities in ScreenToGif

Anastasiia Vorobeva -

mmvergara profile image

I just cloned Threads!

mmvergara -