Linux Command SED, Tutorial with Examples to Relpace String

"sed" is an abbreviation for "stream editor," which, as the name suggests, is used for editing streams. Whether you are writing shell scripts or dealing with STDIN, whenever there is a need for string substitution, copying, or deletion, the sed command is the tool of choice. When sed performs string substitutions, it does not alter the current file. Instead, it keeps the process in a buffer called "pattern space," and after completing the current command, it outputs the results and then moves on to the next command until the file ends.

The sed command is mainly used for non-interactive text processing, which means that you can use it in the command line or within scripts to automate file processing. This makes the sed command a powerful tool for text processing and data transformation.

SED Syntax

sed OPTIONS... [SCRIPT] [INPUTFILE...]

• OPTIONS - Options
• SCRIPT - Action script
• INPUTFILE - Input file

The action script is enclosed in single quotes and supports string substitution using Regex regular expressions.

SED Options

To help with understanding, I have saved the following text – lyrics of PIKO Taro's famous song: PPAP, into a file named sed_example.txt, and will use this file for demonstration. Readers can follow along and practice to reinforce their understanding.

I have a pen, I have an apple
Ah
Apple pen
I have a pen, I have pineapple
Ah
Pineapple pen
Apple pen
Pineapple pen
Ah
Pen Pie Pineapple Apple Pen
Pen Pie Pineapple Apple Pen

Although sed has a plethora of options and commands, in practice, the following are the ones that are commonly used:

• -n：Silent mode.
• -e：Edit in command mode directly. (Can be omitted; more details to follow)
• -f：The script is not entered directly on the command line but is loaded from a specified file.
• -i：Modify the file in-place.

-n (Silent Mode)

By default, all data is generally displayed on the screen. However, if the -n option is added to enable silent mode, only the lines where the p command is used will be displayed.

Let's go through a few examples for better understanding.

Example (a-1)

sed 's/have/had/1' sed_example.txt

The s command is for substitution; it replaces the word "have" with "had". The flag /1 indicates that it should replace the first occurrence. If it is /2, it will replace the second occurrence, and so on.

Output:

This will print the text in the entire file and replace the first occurrence of "have" with "had" in each line.
Note that the s command only replaces text in the output, it does not change the content of the file.

Example (a-2) Adding the -n option

sed -n 's/have/had/2' sed_example.txt

Output:

No output is printed. This is because the -n option is added, activating the silent mode, so no data will be output.

Example (a-3) Adding the -n option along with p command

sed -n 's/have/had/2p' sed_example.txt

Output:

By adding the p command in the flag, only the affected lines will be printed.

-e (Edit)

Example (b-1)

sed -e 's/pen/pencil/' sed_example.txt > sed_output.txt

Output:

This replaces the first occurrence of "pen" with "pencil" in each line of the file, and saves it to the sed_output.txt file.

The -e option can be omitted. The above command is equivalent to:

sed 's/pen/pencil/' sed_example.txt > sed_output.txt

The only difference is that if you want to issue multiple commands in the same line, with -e you do not need to separate them with a semicolon, whereas without -e, you need to add a semicolon ;. Let's look at an example.

Example (b-2)

sed 's/pen/pencil/; s/have/had/' sed_example.txt > sed_output2.txt

This is equivalent to:

sed -e 's/pen/pencil/' -e 's/have/had/' sed_example.txt > sed_output2.txt

Output

Now, you should have a clear understanding of when to use this option ^^

-f (Reading from a Script File)

This option allows reading sed scripts stored in a file. For instance, let's say we put the following text into a file named sed_command.txt.

s/pen/pencil/
s/have/had/

Example (c-1)

sed -f sed_command.txt sed_example.txt > sed_output3.txt

The contents of the file sed_output3.txt would be the same as in Example (b-2), so there's no need to repeat the output here. As we can see, the -f option is suitable for scenarios where there are multiple sed commands to be executed.

-i (Modify File In-place)

This option modifies the file directly instead of printing the output to the screen. It's a commonly used option, often employed in writing automation scripts.

Example (d-1)

In the author's EasyBash project, there is a line inside mysql.sh used for quick MySQL installation.

sudo sed -i "s/bind-address.*/bind-address = 0.0.0.0/" ${cnf_path}

In this practical example, when the user wants to install MySQL using mysql.sh, and issues the command for remote login as follows:

./mysql.sh --version=system --secure=y --remote=y --remote-user=test_user --remote-password=12345678

During the shell processing, sed is used to replace the bind IP setting in my.cnf with a setting that allows connections from any IP address.

Through this example, we can understand that being familiar with the sed command is very helpful in writing automation scripts. There is no longer a need to manually open and edit configuration files.

SED Commands

Commands in sed are actions performed on strings. The convention when using commands with strings is to enclose the command and string in single quotes '. If no string is involved, single quotes are not necessary. Using single quotes is a convention; double quotes " can also be used. However, if the string is a variable, double quotes must be used, otherwise, it would be treated as a plain string.

Here are some of the most commonly used commands in sed:

a (Append)

Inserts a string after a specified line. If no line is specified, the string is inserted after every line.

Syntax

a text

Example (e-1)

sed '1a I have both pen and apple' sed_example.txt

This command inserts the string "I have both pen and apple" after the first line.

Output

Example (e-2)

sed '1,4a I have both pen and apple' sed_example.txt

This command inserts the string "I have both pen and apple" after lines 1 to 4. 1,4 indicates lines 1 to 4.

c (Change)

Replaces a specified line with a replacement string.

Example (e-3)

sed '3c I have both pen and apple' sed_example.txt

This command replaces the entire third line with the string "I have both pen and apple".

Output

Example (e-4)

sed '1,5c I have both pen and apple' sed_example.txt

This command replaces lines 1 to 5 with the string "I have both pen and apple". Note that it doesn't replace each line with the string, but rather it replaces the five lines with a single line, causing the other four lines to disappear.

Output

d (Delete)

Deletes specified lines. It's commonly used with the -i option for removing unnecessary lines when modifying files.

Example (e-5)

sed 1,5d sed_example.txt

This command deletes lines 1 to 5.

Example (e-6)

sed '/pen/d' sed_example.txt

This command deletes all lines that contain the word "pen". Note that the syntax for deleting lines that match a string is as follows:

sed '/text/d'

In this case, the d command is at the end, not at the beginning.

i (Insert)

The usage is the same as the a command, but the difference is that the a (append) command inserts after the specified line, while the i (insert) command inserts before the specified line.

Example (e-7)

sed '1,3i I have both pen and apple' sed_example.txt

This command inserts the string "I have both pen and apple" before each of the lines from line 1 to line 3.

Output

p (Print)

Usually used with the -n option to only print the affected lines. (Refer to Example a-3 above).

s (Substitute)

This is the most commonly used command and supports regular expressions. It is used to substitute strings.

Syntax

s/regexp/replacement/[flags]

This article contains several examples using the s command, you can scroll up to check them.

Common Flags in sed Commands

• [0-9]: A number indicates searching or replacing only the N-th occurrence of the pattern string.
• g: Replace all occurrences
• I: Ignore case
• w: Write the matched result to a file. This gives the same result as using the -n option in conjunction with the p flag. This flag must be placed last if used with other flags.

Practical Examples of sed Commands

Viewing Contents of a File

View lines 1 to 5 of the nginx.conf file.

sed -n '1,5p' /etc/nginx/nginx.conf

Output:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

Changing the Domain Name of a WordPress Website

Step 1: First, export the database

mysqldump -uroot -p terrylin_dev > wordpress.sql

This command uses mysqldump to export the database named terrylin_dev to a file called wordpress.sql.

sed -i 's/terryl.lo/new-domain.com/g' wordpress.sql

This command replaces all occurrences of the domain name terryl.lo with the new domain name new-domain.com in the wordpress.sql file. The g flag is for replacing all occurrences.

Conclusion

sed is an extremely powerful tool with many options and commands. This article is just a basic introduction. In addition to the common options, commands, and flags listed above, the most complete official documentation can be found at the following link:

• sed, a stream editor - gnu.org

This article lists some common use cases for sed, but in practice, there are countless scenarios where it can be used. Therefore, if you want to be proficient in shell scripting, familiarity with the sed command is essential.

Read this article in Traditional Chinese version: Linux SED.