The Stream Editor (sed) is one of the most powerful text-processing tools in Unix and Linux. With sed, you can automate text transformations and streamline tasks, from simple substitutions to complex multi-line edits. Here, we’ll explore the top 10 sed use cases, with practical examples to help you work efficiently from the command line.
1. Basic Text Substitution
A common use of sed is replacing text in files. You can change all occurrences of a word, phrase, or pattern with another—useful for tasks like renaming terms in documents or code.
Example: Replace all occurrences of "cat" with "dog" in pets.txt.
sed 's/cat/dog/g' pets.txt
In this command:
- s denotes the substitution.
-
catis the text to replace. -
dogis the new text. - g ensures all occurrences on each line are replaced. Without it, only the first instance on each line changes.
Why it’s useful: Simple substitutions are essential for text file automation, especially when replacing repetitive terms across long documents or scripts.
2. Replace Text Only on Specific Lines
Sometimes, replacements are needed on specific lines only. With sed, you can specify line numbers or patterns to target specific lines, giving you control over where changes are applied.
Example: Replace "apple" with "orange" only on lines 2 through 4 in fruits.txt.
sed '2,4s/apple/orange/g' fruits.txt
Why it’s useful: This is useful for making targeted changes in files with structured data, like configuration files, where specific lines represent different settings.
3. Deleting Lines by Pattern or Line Number
Cleaning up files or logs often involves removing unwanted lines. With sed, you can delete specific lines based on their numbers or patterns.
Example: Remove all lines containing the word "error" in log.txt.
sed '/error/d' log.txt
Why it’s useful: This helps in keeping logs readable by removing irrelevant or redundant information, particularly in error or debug logs where specific keywords are common.
4. Inserting Text at Specific Locations
Adding content at specific points in a file can be done by inserting lines before or after certain sections. Use i to insert before a line and a to insert after.
Example: Add "Start of Section" before line 1 in document.txt.
sed '1i Start of Section' document.txt
Why it’s useful: Automating the addition of headers, footers, or section markers is valuable when creating structured documents or reports.
5. Replacing the nth Occurrence of a Pattern
In some cases, you may want to replace only the nth instance of a pattern on each line, without affecting other matches.
Example: Replace the third instance of "cat" with "dog" on each line in animals.txt.
sed 's/cat/dog/3' animals.txt
Why it’s useful: This selective replacement is useful in structured text where certain words repeat in specific sequences or when only specific instances need alteration.
6. Print Only Matching Lines
Using -n with p, sed can print only lines matching a specified pattern. This is often faster than using grep in workflows where other sed functions are already in use.
Example: Print only lines containing "error" in logfile.txt.
sed -n '/error/p' logfile.txt
Why it’s useful: This is effective when extracting relevant information from large files, such as pulling all error messages from a log file.
7. Transform Case (Uppercase/Lowercase)
Text case transformations can be especially useful in scripts, either for enforcing consistent formatting or making text processing easier.
Example: Convert all lowercase letters to uppercase in textfile.txt.
sed 's/.*/\U&/' textfile.txt
Why it’s useful: In data processing, transforming cases can ensure consistency, such as enforcing uppercase for case-insensitive matching or processing.
8. Replace Across Multiple Lines
Sometimes, replacements need to span across multiple lines. With sed's multi-line N command, you can match patterns across lines, ideal for structured text with multi-line entries.
Example: Replace "temporary" with "permanent" in all lines between "begin" and "end" patterns.
sed '/begin/,/end/s/temporary/permanent/g' multi_line.txt
Why it’s useful: This approach is essential for handling configuration files or structured logs where entries span multiple lines.
9. Appending Text to the End of Lines
Appending text at the end of each line is common in file formatting tasks, particularly for preparing logs, reports, or structured data files.
Example: Add "checked" at the end of each line in status.txt.
sed 's/$/ checked/' status.txt
Why it’s useful: Appending helps when formatting or tagging lines with identifiers or timestamps for tracking or categorization.
10. Editing Files In-Place
To save changes directly to a file, use -i for in-place editing. This is ideal for batch updates across multiple files without creating temporary copies.
Example: Replace "error" with "issue" across multiple .log files in the current directory.
sed -i 's/error/issue/g' *.log
Why it’s useful: Direct editing saves time in batch operations, like updating configuration files or making systematic replacements in scripts.
Real Life Use Cases
Here are five real-life examples where sed can be particularly useful in managing configurations or modifying files in specific scenarios. These examples include updating hostnames, modifying configuration files, and applying bulk replacements to streamline administrative tasks.
1. Changing Hostnames in Nagios Configuration Files
In a Nagios setup, host and service definition files often contain multiple references to a hostname. When updating the hostname across different files or modifying it in all service definitions, sed can make bulk changes quickly.
Example: Update oldhostname to newhostname across all .cfg files in the Nagios configuration directory.
sed -i 's/oldhostname/newhostname/g' /usr/local/nagios/etc/objects/*.cfg
Explanation: This command updates every occurrence of oldhostname to newhostname in each .cfg file within the directory. This is especially useful when migrating a server or making consistent changes across multiple configurations.
2. Updating SMTP Details in Grafana Configuration (grafana.ini)
Grafana's SMTP settings are configured in the grafana.ini file. If the SMTP server address or port changes, updating this manually can be tedious. Instead, sed can replace these details automatically.
Example: Change SMTP server from old.smtp.com to new.smtp.com in grafana.ini.
sed -i 's/old.smtp.com/new.smtp.com/g' /etc/grafana/grafana.ini
Explanation: This command replaces all instances of the old SMTP server URL with the new one, ensuring email alerts and notifications continue to work without manual edits. Adjusting SMTP ports or authentication details can be similarly streamlined.
3. Modifying Database Credentials in Application Configurations
When applications use hardcoded database credentials in configuration files, such as db_config.php, updating them manually across files is time-consuming. With sed, you can replace the username or password easily.
Example: Update the database username from dbuser_old to dbuser_new in a PHP configuration file.
sed -i 's/dbuser_old/dbuser_new/g' /var/www/html/config/db_config.php
Explanation: This command makes it simple to update credentials across multiple files. Automating the credential update process helps maintain security and consistency, especially when rotating credentials regularly.
4. Replacing IP Addresses in Nginx Configuration Files
In Nginx configurations, updating IP addresses for reverse proxies or server directives can be repetitive. With sed, you can apply IP changes across multiple configuration files at once, avoiding potential typos.
Example: Replace the old IP 192.168.1.10 with the new IP 192.168.1.20 across all Nginx site configuration files.
sed -i 's/192.168.1.10/192.168.1.20/g' /etc/nginx/sites-available/*.conf
Explanation: This is helpful when reassigning internal IPs or restructuring network settings. It ensures consistency in reverse proxy configurations and minimizes manual errors in IP entries.
5. Updating Paths in Shell Scripts After Directory Restructuring
After restructuring directories or changing the file organization, shell scripts may have outdated paths pointing to old locations. Using sed, you can quickly update these paths to match the new directory structure.
Example: Replace all references to /old/path with /new/path in all shell scripts within a specific directory.
sed -i 's|/old/path|/new/path|g' /usr/local/bin/*.sh
Explanation: This command allows for easy updates of file paths within scripts, making it essential when reorganizing directories or migrating to a new file structure. Using | as the delimiter helps when the path contains /.
These examples demonstrate how sed is invaluable for real-life scenarios in IT administration, especially for configuration management, automated updates, and batch modifications. Integrating sedcommands into your workflows can significantly speed up tasks, reduce human error, and make file management more efficient.
Conclusion
The sedcommand is an invaluable tool for anyone who works with text files in Unix or Linux environments. From basic find-and-replace operations to complex pattern matching and multi-line editing, sed enables users to perform efficient text manipulations directly from the command line. Mastering these top 10 sed use cases will streamline your text-processing tasks, save time, and enhance your command-line proficiency.
🔗 Support my Work
▶️ Support by Subscribing my YouTube
▶️ Explore more open-source tutorials on my website
▶️ Follow me on X
☕ Buy me a Coffee
Top comments (3)
Nice article.
It would be good to mention at the beginning that sed is a filter, so all your initial examples output to the terminal which is not usually what you want.
I looked at my ~/bin and almost every sed invocation uses piped data into or out of a sed into something else or using it in a bash command substitution to create a new value in a variable.
That's where sed really shines. These uses make for more complicated examples, but they get at why to use sed more than just how to use it.
Another thing to keep in mind is that, for the most part, regular expressions are write-only code, so if you spend any length of time debugging one, it's usually much less painful to rewrite it in awk where you can separate out steps and print partial results for debugging. This is particularly true for multi-line or context dependent changes which are really hard in sed. This is even more important when maintaining code later when you (or your successors) might not remember how a particular regex works.
Here's an example.
Remove left-padded zeros from a numeric string:
next=$(sed -re 's/^0*([0-9]+)/\1/' <<< ${next})Hey Joe, this is quite an amazing review. People like you make us write more quality content.
This old-as-the-Internet document is a real treasure trove: edoras.sdsu.edu/doc/sed-oneliners....