Assigning values to variables in a pipeline. Using the xargs command
What is needed?
You should become familiar with the command line as well as basic programming concepts. Although this is not a programming tutorial, it explains (or at least attempts to explain) many basic concepts.
Use of this document
This document may be necessary in the following situations:
You have ideas related to programming, and there is a need to carry out the process of coding some shell scripts.
Your programming ideas are not specific enough and require additional guidance.
Would you like to take a look at some shell scripts and comments as an example for creating your own?
You are migrating from DOS/Windows (or have already done so) and want to create files batch processing("batch").
You are a complete nerd and read any how-to that comes to hand.
The simplest scripts
This HOW-TO attempts to provide you with some guidelines for shell programming, based only on examples.
In this section you will find small scripts that will probably be useful to you when mastering some techniques.
Traditional "hello world" script
#!/bin/bash echo Hello World!
This script contains only two lines. The first tells the system what program is being used to run the file.
The second line is the only action this script performs, printing "Hello world" to the terminal.
If you receive something like ./hello.sh: Command not found. , then perhaps the first line "#!/bin/bash" is wrong; run whereis bash or look at finding bash to figure out what this line should be.
Simple backup script
#!/bin/bash
tar -cZf /var/my-backup.tgz /home/me/
In this script, instead of printing a message on the terminal, we create a tar archive of the user's home directory. The script is NOT intended for practical use. Next in this document a more efficient script will be presented Reserve copy.
All About Redirection Theory and Quick View
There are 3 file descriptors: stdin - standard input, stdout - standard output and stderr - standard error.
Your main features:
redirect stdout to file
redirect stderr to file
redirect stdout to stderr
redirect stderr to stdout
redirect stderr and stdout to file
redirect stderr and stdout to stdout
redirect stderr and stdout to stderr
1 means stdout and 2 means stderr. A quick note for further understanding: with the less command, you can view both stdout, which remains in the buffer, and stderr, which is printed to the screen. However, it is erased when you attempt to "browse" the buffer.
Example: stdout to file
This action writes the program's standard output to a file.
ls -l > ls-l.txt
This creates a file called "ls-l.txt". It will contain everything that you would see if you simply ran the command "ls -l". 3.3 Example: stderr to file
This action writes the program's standard error stream to a file.
grep da * 2> grep-errors.txt
This creates a file called "grep-errors.txt". It will contain the standard error portion of the "grep;da;*" command output. 3.4 Example: stdout to stderr
This action writes the program's standard output to the same file as the standard error stream.
grep da * 1>&2
Here the standard output of the command is sent to standard error. You can see it different ways. 3.5 Sample: stderr 2 stdout
This action writes the program's standard error stream to the same place as standard output.
grep * 2>&1
Here the standard error stream of the command is sent to standard output; If you pipe the result (|) to less, you will see that lines that would normally be lost (as written to standard error) are saved in this case (as they are on standard output). 3.6 Example: stderr and stdout to file
This action puts all of the program's output into a file. This is a good option for cron jobs: if you want the command to run completely silently.
rm -f $(find / -name core) &> /dev/null
This (assuming for cron) deletes any file called "core" in any directory. Remember that you should be completely sure of what the command does if you want to overwrite its output. 4. Conveyors
This section explains in a fairly simple and practical way how conveyors should be used and why you might need them.
What is it and why should you use it?
Pipelines give you the ability to use (the author is convinced that this is quite simple) the output of one program as the input of another.
Example: a simple pipeline with sed
This is a very simple way to use conveyors.
ls -l | sed -e "s//u/g"
What happens here is that the command ls;-l is initially executed, and its output, instead of being displayed on the screen, is sent to the sed program, which in turn prints what it should on the screen. 4.3 Example: alternative for ls;-l;*.txt
Perhaps it is much more the hard way, than ls;-l;*.txt, but it is provided here only to illustrate the work with pipelines, and not to decide the choice between these two listing methods.
ls -l | grep "\.txt$"
Here, the output of ls -l is sent to grep, which in turn prints the lines matching the regular expression "\.txt$". 5. Variables
You can use variables in the same way as in any programming language. There are no data types. A variable in bash can be a number, a character, or a string of characters.
You shouldn't declare a variable. In fact, assigning a value to its pointer already creates it.
Example: "Hello World!" using variables
#!/bin/bash
STR="Hello World!"
echo $STR
The second line creates a variable called STR and assigns it the string value "Hello World!". The VALUE of this variable is then extracted by adding a leading "$" sign. Please remember (try hard) that if you do not use the "$" sign, the program output may be different. Probably not the one you need.
Example: very simple backup script (more efficient)
#!/bin/bash
OF=/var/my-backup-$(date +%Y%m%d).tgz #OF - Output File - output file
tar -cZf $OF /home/me/
This script introduces another concept. First of all, you should deal with the second line. Pay attention to the expression "$(date +%Y%m%d)". If you run this script, you will notice that it executes the command inside the parentheses, intercepting its output.
Note that in this script, the output file name will change daily based on the format of the date;(+%Y%m%d) command key. You can replace this with a different format assignment.
Other examples:
echo ls
echo $(ls)
Local variables
Local variables can be created by using the local keyword.
#!/bin/bash
HELLO=Hello
function hello(
local HELLO=World
echo $HELLO
}
echo $HELLO
hello
echo $HELLO
This example should be sufficient to show how local variables can be used.
Conditional statements
Conditional statements give you the ability to decide whether to perform an action or not; the decision is made when calculating the value of the expression.
Just a theory
Exists a large number of forms of conditional statements. The elementary form is an if expression then statement, where the "statement" is executed only if the "expression" evaluates to "true". "2<1" - это выражение, имеющее значение "ложь", в то время как "2>1" - "true".
There are other forms of conditional statements such as: if expression then statement1 else statement2. Here "operator1" is executed if "expression" is true; otherwise, "statement2" is executed.
Another form of conditional statements is: if expression1 then operator1 else if expression2 then operator2 else operator3. This form only adds the sequence "ELSE IF "expression2" THEN "statement2"", causing "statement2" to be executed if "expression2" evaluates to "true". Everything else corresponds to your idea of this (see previous forms).
A few words about the syntax:
The basic construction of an "if" statement in bash looks like this:
if [expression];
then
code if "expression" is true.
fi
Example: a basic example of a conditional statement if;..;then
#!/bin/bash
if ["foo" = "foo" ]; then
fi
If the expression inside the square brackets is true, then the code to be executed is after the word "then" and before the word "fi", which indicates the end of the code to be executed when the condition is met.
Example: a basic example of a conditional statement if;..;then;...;else
#!/bin/bash
if ["foo" = "foo" ]; then
echo expression evaluates to true
else
fi
Example: Conditional Statements with Variables
#!/bin/bash
T1="foo"
T2="bar"
if [ "$T1" = "$T2" ]; then
echo expression evaluates to true
else
echo expression evaluates to false
fi
for, while and until loops
In this section, you will become familiar with for, while, and until loops.
The for loop is slightly different from its counterparts in other programming languages. First of all, it gives you the opportunity to perform consistent actions above the "words" in the line.
The while loop executes a piece of code if the expression being tested is true; and stops if it is false (or an explicitly specified loop interrupt is encountered within the executing code).
The until loop is almost identical to the while loop. The only difference is that the code is executed if the expression being tested is false.
If you assume that while and until are very similar, you are right.
Example for loop
#!/bin/bash
for i in $(ls); do
echo item: $i
done
In the second line we represent i as a variable that receives the various values contained in $(;ls;).
If necessary, the third line could be longer; or there could be several lines before done (4th line).
"done" (4th line) indicates that the code using $i is ending and $i is given a new value.
This script is not intended to be of great importance. A more useful use of the for loop would be to use it to select only certain files in the previous example.
C-like for
fiesh suggested adding this loop form. This for loop, most similar to for in languages C, Perl, etc.
#!/bin/bash
for i in `seq 1 10`;
do
echo $i
done
Example while loop:
#!/bin/bash
COUNTER=0
while [ $COUNTER -lt 10 ]; do
echo The counter is $COUNTER
let COUNTER=COUNTER+1
done
This script “emulates” the well-known (in C, Pascal, perl, etc.) structure “for”.
Example until loop:
#!/bin/bash
COUNTER=20
until [ $COUNTER -lt 10 ]; do
echo COUNTER $COUNTER
let COUNTER-=1
done
Functions
Just like any other programming language, you can use functions to group pieces of code together in a more logical way, as well as practice the magic art of recursion.
A function declaration is just a function my_func ( my_code ) entry.
Calling a function is carried out in the same way as calling other programs. You just write her name.
Example functions
#!/bin/bash
function quit (
exit
}
function hello(
echo Hello!
}
hello
quit
echo foo
Lines 2-4 contain the "quit" function. Lines 5-7 contain the "hello" function. If you don't understand the process performed by this script, try it out!
It should be noted that it is not necessary to declare functions in any particular order.
If you run the script, notice that the "hello" function is called first, and then the "quit" function. As for the program, it never reaches the 10th line.
Example of functions with parameters
#!/bin/bash
function quit (
exit
}
function e (
echo $1
}
Hello
e World
quit
echo foo
This script is almost identical to the previous one. The main difference is the "e" function. It prints the very first argument it receives. Arguments in functions are processed in the same way as arguments passed to the script.
User Interfaces
Using select to create simple menus
#!/bin/bash
OPTIONS="Hello Quit"
select opt in $OPTIONS; do
if [ "$opt" = "Quit" ]; then
echo done
exit
elif [ "$opt" = "Hello" ]; then
echo Hello World
else
clear
echo bad option
fi
done
If you run this script, you will see that it is a programmer's dream of a text-based menu. You'll probably notice that this is very similar to the "for" construct, but instead of looping through each "word" in $OPTIONS, the program polls the user.
Using the Command Line
#!/bin/bash
if [ -z "$1" ]; then
echo use: $0 directory
exit
fi
SRCD=$1 #SRCD - SouRCe Directory - source directory
tar -cZf $TGTD$OF $SRCD
It should be clear to you what this script does. The expression in the first conditional statement checks whether the program received an argument ($1). If not, it terminates the program, presenting the user with a small error message. Remaining on this moment part of the script is obviously understandable.
Miscellaneous
Reading user input with read
In some cases, it may be necessary to ask the user to enter something. Exist various ways doing this. One way is the following:
#!/bin/bash
echo Please enter your name
read NAME
echo "Hello $NAME!"
Alternatively, you can get multiple values at once using read. The following example explains this:
#!/bin/bash
echo "Please enter your first and last name"
read FN LN #FN - First Name - name; LN - Last Name - surname
echo "Hi! $LN, $FN !"
Arithmetic calculations
At the command line (or shell), try typing the following:
echo;1;+;1
If you expect to see "2", you will be disappointed. What should you do if you need BASH to perform calculations on your numbers? The solution is this:
echo;$((1+1))
As a result, the conclusion will be more "logical". This notation is used to evaluate arithmetic expressions. You can also do it like this:
echo;$
If you need to use fractions or more complex mathematics, you can use bc to evaluate arithmetic expressions.
When the author ran "echo;$" in command shell, it returned the value 0. This is because when bash responds, it only uses integer values. If you run "echo;3/4|bc;-l", the shell will return the correct value 0.75.
bash search
From a message from mike (see "Acknowledgments" section):
You always use #!/bin/bash .. Could you give an example of how to discover where bash is located.
It is preferable to use "locate bash", but locate is not available on all machines.
"find ./ -name bash" from root directory usually works.
You can check the following locations:
ls -l /bin/bash
ls -l /sbin/bash
ls -l /usr/local/bin/bash
ls -l /usr/bin/bash
ls -l /usr/sbin/bash
ls -l /usr/local/sbin/bash
(the author is unable to immediately think of any other directory... He found bash in most of these places on various systems).
You can also try "which bash".
Getting the program's return value
In bash, the program's return value is stored in a special variable called $?.
This example illustrates how to intercept the return value of a program; the author assumed that the dada directory does not exist (this was also suggested by mike).
#!/bin/bash
cd /dada &> /dev/null
echo rv: $?
cd $(pwd) &> /dev/null
echo rv: $?
Intercepting command output
This small script presents all tables from all databases (assuming you have MySQL installed). Additionally, you should think about ways to convert the "mysql" command to use a suitable username and password.
#!/bin/bash
DBS=`mysql -uroot -e"show databases"`
for b in $DBS ;
do
mysql -uroot -e"show tables from $b"
done
Multiple source files
You can run multiple files using the source command.
TO-DO__
11. Tables
11.1 String comparison operators
(1) s1 = s2
(2) s1 != s2
(3) s1< s2
(4) s1 > s2
(5) -n s1
(6) -z s1
(1) s1 coincides with s2
(2) s1 does not coincide with s2
(3) s1 alphabetically precedes s2 (according to the current locale)
(4) s1 comes alphabetically after s2 (according to current locale)
(5) s1 has no null value(contains one character or more)
(6) s1 has zero value
11.2 Examples of string comparisons
Comparing two strings.
#!/bin/bash
S1="string"
S2="String"
if [ $S1=$S2 ];
then
echo "S1("$S1") is not equal to S2("$S2")"
fi
if [ $S1=$S1 ];
then
echo "S1("$S1") is equal to S1("$S1")"
fi
At this point, the author feels it necessary to quote a remark from an email received from Andreas Beck regarding the use of if [ $1 = $2 ].
This is not a good idea, since if either $S1 or $S2 is empty line, You'll get syntax error. It would be more acceptable to use x$1;=;x$2 or "$1";=;"$2" .
11.3 Arithmetic operators
+
-
*
% (remainder)
11.4 Arithmetic comparison operators
-lt (<)
-gt (>)
-le (<=)
-ge (>=)
-eq (==)
-ne (!=)
C programmers simply need to select the operator that matches the selected operator in the parentheses.
Useful commands
This section was rewritten by Kees (see Acknowledgments section).
Some of these commands practically contain full-fledged command languages. Only the basics of such commands are explained here. For more detailed information Carefully review the man pages for each command.
sed (stream editor)
Sed is a non-interactive editor. Instead of changing the file by moving the cursor on the screen, you should use a sed editing instructions script along with the name of the file you are editing. You can also think of sed as a filter. Look at some examples:
$sed "s/to_be_replaced/replaced/g" /tmp/dummy
Sed replaces the string "to_be_replaced" with the string "replaced" by reading the /tmp/dummy file. The result is sent to standard output (usually the console), but you can also add ">;capture" to the above line to have sed send the output to the "capture" file.
$sed 12, 18d /tmp/dummy
Sed displays all lines except lines 12 through 18. Original file is not changed by this command.
awk (data file manipulation, text retrieval and processing)
There are a large number of implementations of the AWK programming language (the most common interpreters are gawk from the GNU project and the "new awk" mawk.) The principle is quite simple: AWK is in search of a pattern; For each matching template, some action is performed.
The author has recreated the dummy file containing the following lines:
"test123
test
tteesstt"
$awk "/test/ (print)" /tmp/dummy
test123
test
The pattern AWK looks for is "test", and the action AWK takes when it encounters a line in /tmp/dummy with the substring "test" is "print".
$awk "/test/ (i=i+1) END (print i)" /tmp/dummy
If you are searching for multiple patterns, replace the text between the quotes with "-f;file.awk". In this case, you can write all the templates and actions in the file "file.awk".
grep (prints lines matching the search pattern)
We've looked at several grep commands in previous chapters that display lines that match a pattern. However, grep can do much more.
$grep "look for this" /var/log/messages -c
The string "look for this" was found 12 times in /var/log/messages.
wc (counts lines, words and bytes)
In the following example, you will notice that the output is not what we expect. In this case, the dummy file contains next text:
"bash introduction
howto test file"
$wc --words --lines --bytes /tmp/dummy
2 5 34 /tmp/dummy
wc doesn't care about the order of the parameters. It always outputs them in the standard order:<число;строк><число;слов><число;байтов><имя;файла>.
sort (sorts the lines of a text file)
In this case, the dummy file contains the following text:
"b
c
a"
$sort /tmp/dummy
The output looks like this:
a
b
c
Commands shouldn't be so simple :-)
bc (computational programming language)
bc performs calculations from the command line (input from a file, but not via redirection or pipelining) as well as from the user interface. The following example shows some commands. Note that the author used bc with the -q option to suppress the prompt message.
$bc -q
1 == 5
0
0.05 == 0.05
1
5 != 5
0
2 ^ 8
256
sqrt(9)
3
while (i != 9) (
i = i + 1;
print i
}
123456789
quit
tput (initializes a terminal or queries the terminfo database)
A small illustration of tput's capabilities:
$tput cup 10 4
The command prompt will appear at coordinates (y10,x4).
$tput reset
The screen clears and the prompt appears at (y1,x1). Note that (y0,x0) is the top left corner.
$tputcols
80 Displays the possible number of characters in the x-direction.
It is strongly recommended to be familiar with these programs (at a minimum). Exists great amount small programs, which give you the opportunity to do some real magic on the command line.
[Some examples were taken from man pages or FAQ.]
More scripts
Apply the command to all files in a directory.
Example: very simple backup script (more efficient)
#!/bin/bash
SRCD="/home/" #SRCD - SouRCe Directory - source directory
TGTD="/var/backups/" #TGTD - TarGeT Directory - final directory
OF=home-$(date +%Y%m%d).tgz #OF - Output File - output file
tar -cZf $TGTD$OF $SRCD
File renaming program
#!/bin/sh
# renna: rename multiple files using special rules
# Author - felix hudson Jan - 2000
#First of all, look at the different "modes" this program has.
#If the first argument ($1) is suitable, we execute this part
#programs and leave.
# Check for the possibility of adding a prefix.
if [ $1 = p ]; then
#Now we move on from the mode variable ($1) and the prefix ($2)
prefix=$2 ; shift ; shift
# Must check if at least one file is specified.
# Otherwise, it's better to do nothing than rename non-existent
# files!!
if [$1 = ]; then
exit 0
fi
# This for loop processes all the files we have specified
# program.
# It does one rename per file.
for file in $*
do
mv $(file) $prefix$file
done
#After this the program is exited.
exit 0
fi
# Check for the condition of adding a suffix.
# Otherwise, this part is virtually identical to the previous section;
# please see the comments contained therein.
if [ $1 = s ]; then
suffix=$2 ; shift ; shift
if [$1 = ]; then
echo "no files specified"
exit 0
fi
for file in $*
do
mv $(file) $file$suffix
done
exit 0
fi
# Check for renaming condition with replacement.
if [ $1 = r ]; then
shift
# For security reasons, the author included this part so as not to damage any file if the user
# didn't define what to do:
if [ $# -lt 3 ] ; then
echo "Error; correct input: renna r [expression] [replacement] files..."
exit 0
fi
# Let's look at other information
OLD=$1 ; NEW=$2 ; shift ; shift
# This for loop sequentially goes through all the files that we
# assigned to the program.
# It does one rename per file using the "sed" program.
# This simple program from the command line, which parses the standard
# input and replace regular expression to a given line.
# Here we give sed a filename (as standard input) and replace
# required text.
for file in $*
do
new=`echo $(file) | sed s/$(OLD)/$(NEW)/g`
mv $(file) $new
done
exit 0
fi
# If we reach this line, it means that the program has been given
# invalid parameters. In this regard, it should be explained to the user how to
# use
echo "use:"
echo "renna p [prefix] files.."
echo "renna s [suffix] files.."
echo "renna r [expression] [replacement] files.."
exit 0
#done!
File renaming program (simple)
#!/bin/bash
#renames.sh
# simple renaming program
criteria=$1
re_match=$2
replace=$3
For i in $(ls *$criteria*);
do
src=$i
tgt=$(echo $i | sed -e "s/$re_match/$replace/")
mv $src $tgt
done
If what happens is different from what you expect (debugging)
How can I call BASH?
It would be nice to add in the first line
#!/bin/bash -x
This will produce some interesting output information.
About the document
You should not hesitate to make corrections, additions or anything else if, in your opinion, it should be present in this document. The author will try to update it whenever possible.
The for-in operator is intended for sequential access to the values listed in the list. Each value in turn in the list is assigned to a variable. The syntax is as follows:
for variable in value_list do commands doneLet's consider small example:
#!/bin/bash for i in 0 1 2 3 4 #we will alternately assign values from 0 to 4 inclusive to the variable $i do echo "Console number is $i ">> /dev/pts/$i #Write the line "Console number is $i" to the file /dev/pts/$i (virtual terminal file) done #loop finished exit 0After running the example, a line with its number will appear in the first 5 virtual consoles (terminals). Values from the list are alternately substituted into the variable $i and the value of this variable is worked in a loop.
Cycles. while loop
The while loop is more complex than the for-in loop and is used to repeat commands as long as some expression is true (return code = 0). The operator syntax is as follows:
while expression or command returning the return code of the do command doneLet's look at the following example of how the loop works:
#!/bin/bash again =yes #assign the value "yes" to the variable again while [" $again"= "yes" ] #We will loop until $again is equal to "yes" do echo "Please enter a name:" read name echo "The name you entered is $name" echo "Do you wish to continue?"read again done echo "Bye-Bye"
And now the result of the script:
Ite@ite-desktop:~$ ./bash2_primer1.sh Please enter a name: ite The name you entered is ite Do you wish to continue? yes Please enter a name: mihail The name you entered is mihail Do you wish to continue? no Bye-Bye
As you can see, the loop runs until we enter something other than “yes”. Between do and done you can describe any structures, operators, etc., all of them will be executed in a loop. But you should be careful with this loop, if you run any command in it without changing the expression variable, you can get caught in an endless loop.
Now about the truth condition. After the while, as in the if-then-else conditional statement, you can insert any expression or command that returns the return code, and the loop will be executed until the return code = 0! The operator "[" is an analogue of the test command, which checks the truth of the condition that was passed to it. Let's look at another example, I took it from the book Advanced Bash scripting. I really liked it Smile, but I simplified it a little. In this example, we'll introduce another type of UNTIL-DO loop. This is almost a complete analogue WHILE-DO cycle
, only executed while some expression is false.
Here's an example: #!/bin/bash echo"Enter numerator: " read dividend echo"Enter denominator: " read divisor dnd =$dividend #we will change the variables dividend and divisor,#let's save their knowledge in other variables, because... they give us #will need dvs =$divisor remainder =1 until [ "$remainder " -eq 0 ] do let"remainder = dividend % divisor" dividend =$divisor divisor =$remainder done echo "GCD of numbers$dnd and $dvs = $dividend "
Ite@ite-desktop:~$ ./bash2_primer3.sh Enter numerator: 100 Enter denominator: 90 GCD of 100 and 90 = 10
Mathematical operations
let command.
The let command produces arithmetic operations over numbers and variables.
Let's look at a small example in which we perform some calculations on the entered numbers:
#!/bin/bash echo "Enter a: " read a echo "Enter b: " read b let "c = a + b" #addition echo "a+b= $c" let "c = a / b" #division echo "a/b= $c" let "c<<= 2" #shifts c 2 bits to the left echo "c after shift by 2 bits: $c " let "c = a % b" # finds the remainder of a divided by b echo " $a / $b . remainder: $c "Execution result:
Ite@ite-desktop:~$ ./bash2_primer2.sh Enter a: 123 Enter b: 12 a+b= 135 a/b= 10 c after shift by 2 digits: 40 123 / 12. remainder: 3
Well, as you can see, there is nothing complicated, the list of mathematical operations is standard:
Addition
- - subtraction
* - multiplication
/ - division
** - exponentiation
% - modulus (modulo division), remainder of division
let allows you to use abbreviations for arithmetic instructions, thereby reducing the number of variables used.
For example: a = a+b is equivalent to a +=b, etc.
Working with external programs when writing shell scripts
First, some useful theory.
Stream redirection
Bash (like many other shells) has built-in file descriptors: 0 (stdin), 1 (stdout), 2 (stderr).
stdout - Standard output. Everything that programs output goes here
stdin - Standard input. This is all that the user types in the console
stderr - Standard error output.
For operations with these handles, there are special characters: > (output redirection),< (перенаправление ввода). Оперировать ими не сложно. Например:
cat /dev/random > /dev/nullredirect the output of cat /dev/random to /dev/null (a completely useless operation) or
ls -la > listingwrite the contents of the current directory to the listing file (more useful)
If there is a need to append to a file (when using ">" it is replaced), you must use ">>" instead of ">"
sudo< my_passwordafter asking sudo for the password, it will be taken from the my_password file, as if you had entered it from the keyboard.
If you need to write to a file only errors that could occur while running the program, you can use:
./ program_with_error 2 > error_filethe number 2 before ">" means that you need to redirect everything that ends up in descriptor 2 (stderr).
If you need to force stderr to write to stdout, then this can be done as follows. way:
./ program_with_error 2 >& 1the symbol "&" means a pointer to descriptor 1(stdout)
(By default, stderr writes to the console in which the user is working (or rather, writes to the display)).
2. Conveyors
The pipeline is a very powerful tool for working with the Bash console. The syntax is simple:
team1 | command 2 - means that the output of command 1 will be passed as input to command 2
Pipelines can be grouped into chains and output using redirection to a file, for example:
ls -la |grep "hash" | sort > sortilg_list
The output of the ls -la command is passed to the grep command, which selects all lines that contain the word hash, and passes it to the sort command, which writes the result to the sorting_list file. Everything is quite clear and simple.
Most often, Bash scripts are used to automate some routine operations in the console, hence sometimes there is a need to process the stdout of one command and transfer it to stdin to another command, while the result of executing one command must be processed in some way. In this section I will try to explain the basic principles of working with external commands inside a script. I think that I have given enough examples and now I can write only the main points.
1. Passing output to a variable
To record the output of a command into a variable, simply enclose the command in `` quotes, for example
A = ` echo "qwerty" ` echo $a
Result: qwerty
However, if you want to store a list of directories in a variable, you must properly process the result to place the data in the variable. Let's look at a small example:LIST =` find / svn/ -type d 2>/ dev/ null|
As you can see, everything is not difficult, just understand the principle and write a couple of your own scripts. In conclusion of the article, I would like to wish you good luck in learning BASH and Linux in general. Criticism, as usual, is welcome. The next article may be devoted to the use of programs such as sed, awk.
Learning Linux, 101
Streams, program channels and redirects
Learning the Basics of Linux Pipelines
Content Series:
Short review
In this article, you will learn the basic techniques for redirecting standard I/O streams in Linux. You will learn:
- Redirect standard input/output streams: standard input, standard output, and standard error.
- Direct the output of one command to the input of another command.
- Send output to the standard output device (stdout) and to a file simultaneously.
- Use the output of a command as arguments to another command.
This article will help you prepare to take the LPI 101 Entry Level Administrator (LPIC-1) exam and contains material from Objective 103.4 of Topic 103. The objective has a weight of 4.
About this series
This series of articles will help you master the tasks of administering the Linux operating system. You can also use the material in these articles to prepare for.
To view descriptions of articles in this series and get links to them, please refer to our. This list is continually updated with new articles as they become available and contains the most current (as of April 2009) LPIC-1 certification exam objectives. If an article is missing from the list, you can find an earlier version that meets previous LPIC-1 goals (prior to April 2009) by referring to our .
The necessary conditions
To get the most out of our articles, you need to have a basic knowledge of Linux and have a working Linux computer that can run all the commands you encounter. Sometimes different versions of programs produce results differently, so the contents of the listings and figures may differ from what you see on your computer.
Preparing to Run the Examples
How to contact Ian
Ian is one of our most popular and prolific authors. Check out (EN) published on developerWorks. You can find contact information at and connect with him and other My developerWorks contributors and contributors.
To run the examples in this article, we will use some of the files created earlier in the " " article. If you haven't read this article or saved these files, don't worry! Let's start by creating a new directory lpi103-4 and all the necessary files. To do this, open a text window and navigate to your home directory. Copy the contents of Listing 1 into the text box; as a result of executing the commands, the lpi103-4 subdirectory and all the necessary files in it will be created in your home directory, which we will use in our examples.
Listing 1. Creating the files needed for this article's examples
mkdir -p lpi103-4 && cd lpi103-4 && ( echo -e "1 apple\n2 pear\n3 banana" > text1 echo -e "9\tplum\n3\tbanana\n10\tapple" > text2 echo "This is a sentence. " !#:* !#:1->text3 split -l 2 text1 split -b 17 text2 y; )Your window should look like Listing 2, and your current working directory should be the newly created lpi103-4 directory.
Listing 2. Results of creating the necessary files
$ mkdir -p lpi103-4 && cd lpi103-4 && ( > echo -e "1 apple\n2 pear\n3 banana" > text1 > echo -e "9\tplum\n3\tbanana\n10\tapple" > text2 > echo "This is a sentence." !#:* !#:1->text3echo "This is a sentence." "This is a sentence." "This is a sentence.">text3 > split -l 2 text1 > split -b 17 text2 y; ) $Redirecting standard input/output
A Linux shell, such as Bash, receives input and sends output in the form of sequences or streams characters. Any character is independent of either previous or subsequent characters. Symbols are not organized into structured entries or fixed-size blocks. Streams are accessed using I/O mechanisms regardless of where the character streams come from or are sent to (a file, keyboard, window, screen, or other I/O device). Linux shells use three standard I/O streams, each of which is assigned a specific file descriptor.
- stdout – standard output, displays command output and has handle 1.
- stderr – standard error stream, displays command errors and has descriptor 2.
- stdin – standard input, passes input to commands and has handle 0.
Input streams provide input (usually from the keyboard) to commands. Output streams provide printing of text characters, typically to a terminal. The terminal was originally an ASCII printing device or display terminal, but now it is usually just a window on the computer desktop.
If you have already read the "" guide, then some of the material in this article will be familiar to you.
Output redirection
There are two ways to redirect output to a file:
n> redirects output from a file descriptor n to file. You must have write permissions to the file. If the file does not exist, it will be created. If the file exists, then all its contents are usually destroyed without any warning. n>> also redirects output from the file descriptor n to file. You must also have write permissions to the file. If the file does not exist, it will be created. If the file exists, the output is appended to its contents.Symbol n in operators n> or n>> is file descriptor. If it is not specified, the standard output device is assumed to be used. Listing 3 demonstrates the redirection operation to separate the standard output and standard error of the ls command, using files that were created earlier in the lpi103-4 directory. Adding command output to existing files is also demonstrated.
Listing 3. Output redirection
$ ls x* z* ls: cannot access z*: No such file or directory xaa xab $ ls x* z* >stdout.txt 2>stderr.txt $ ls w* y* ls: cannot access w*: No such file or directory yaa yab $ ls w* y* >>stdout.txt 2>>stderr.txt $ cat stdout.txt xaa xab yaa yab $ cat stderr.txt ls: cannot access z*: No such file or directory ls: cannot access w*: No such file or directoryWe've already said that redirecting output using the n> operator usually results in overwriting existing files. You can control this property using the noclobber option of the set built-in command. If this option is defined, you can override it using the n>| operator, as shown in Listing 4.
Listing 4. Redirecting output using the noclobber option
$ set -o noclobber $ ls x* z* >stdout.txt 2>stderr.txt -bash: stdout.txt: cannot overwrite existing file $ ls x* z* >|stdout.txt 2>|stderr.txt $ cat stdout.txt xaa xab $ cat stderr.txt ls: cannot access z*: No such file or directory $ set +o noclobber #restore original noclobber settingSometimes you may want to redirect both standard output and standard error to a file. This is often used in automated processes or background jobs so that the results of the work can be reviewed later. To redirect standard output and standard error to the same location, use the &> or &>> operator. Alternative option is to redirect the file descriptor n and then the file descriptor m to the same place using the construction m>&n or m>>&n. In this case, the order in which the threads are redirected is important. For example, the command
command 2>&1 >output.txt
it's not the same as a command
command >output.txt 2>&1
In the first case, the error stream stderr is redirected to the current location of the stdout stream, and then the stdout stream is redirected to the output.txt file; however, the second redirection only affects stdout and not stderr. In the second case, the stderr stream is redirected to the current location of the stdout stream, that is, to the output.txt file. These redirections are illustrated in Listing 5. Notice in the last command that the standard output was redirected after the standard error stream, and as a result, the error stream continues to be output to the terminal window.
Listing 5. Redirecting two streams to one file
$ ls x* z* &>output.txt $ cat output.txt ls: cannot access z*: No such file or directory xaa xab $ ls x* z* >output.txt 2>&1 $ cat output.txt ls: cannot access z*: No such file or directory xaa xab $ ls x* z* 2>&1 >output.txt # stderr does not go to output.txt ls: cannot access z*: No such file or directory $ cat output. txt xaa xabIn other situations, you may want to ignore standard output or standard error entirely. To do this, redirect the corresponding stream to the empty file /dev/null. Listing 6 shows how to ignore the error stream from the ls command and how to use the cat command to verify that the file /dev/null is in fact empty.
Listing 6. Ignoring standard error by using /dev/null
$ ls x* z* 2>/dev/null xaa xab $ cat /dev/nullInput redirection
Just like we can redirect stdout and stderr, we can redirect stdin from a file using the operator<. Если вы прочли руководство " ", то должны помнить, что в разделе была использована команда tr для замены пробелов в файле text1 на символы табуляции. В том примере мы использовали вывод команды cat чтобы создать стандартный поток ввода для команды tr . Теперь для преобразования пробелов в символы табуляции вместо бесполезного вызова команды cat мы можем использовать перенаправление ввода, как показано в листинге 7.
Listing 7. Input redirection
$ tr " " "\t"Command interpreters, including bash, implement the concept here-document, which is one of the ways to redirect input. It uses the design<< и какое-либо слово, например END, являющееся маркером, или сигнальной меткой, означающей конец ввода. Эта концепция продемонстрирована в листинге 8.
Listing 8. Redirecting input using the here-document concept
$sort -k2<But why can’t you just type the command sort -k2 , enter the data and press the combination Ctrl-d, indicating the end of the input? Of course, you could run this command, but then you wouldn't know about the here-document concept, which is very common in shell scripts (where there is no other way to specify which lines should be accepted as input). Since tabs are widely used in scripts to align text and make them easier to read, there is another technique for using the here-document concept. When using the operator<<- вместо оператора << начальные символы табуляции удаляются.
In Listing 9, we used command substitution to create a tab character, and then created a small shell script containing two cat commands, each of which reads data from the here-document block. Notice that we used the word END to signal the here-document block that we are reading from the terminal. If we were to use this same word in our script, our input would end prematurely. Therefore, instead of the word END, we use the word EOF in the script. Once our script is created, we use the command. (dot) to run it in the context of the current shell.
Listing 9. Redirecting input using the here-document concept
$ ht=$(echo -en "\t") $ cat<In future articles in this series, you'll learn more about command substitution and scripting. Links to all articles in this series can be found in.
Creating pipelines
Using the xargs command
The xargs command reads data from the standard input device and then constructs and executes commands that take the received input as parameters. If no command is specified, the echo command is used. Listing 12 shows a simple example of using our text1 file, which contains three lines of two words each.
Listing 12. Using the xargs command
$ cat text1 1 apple 2 pear 3 banana $ xargsWhy then does xargs output contain only one line? By default, xargs splits the input if it encounters delimiter characters, and each resulting fragment becomes a separate parameter. However, when xargs builds a command, it is passed as many parameters as possible at one time. This behavior can be changed using the –n or --max-args option. Listing 13 shows an example of both options; an explicit call to the echo command was also made for use with xargs.
Listing 13. Using xargs and echo commands
$xargsIf the input data contains spaces, but they are enclosed in single or double quotes(or spaces are represented as escape sequences using backslashes), then xargs will not break them into separate parts. This is shown in Listing 14.
Listing 14. Using the xargs command and quotes
$ echo ""4 plum"" | cat text1 - 1 apple 2 pear 3 banana "4 plum" $ echo ""4 plum"" | cat text1 - | xargs -n 1 1 apple 2 pear 3 banana 4 plumUntil now, all arguments have been added to the end of the command. If you need other optional arguments to be added after them, use the -I option to specify a replacement string. At the point in the command called via xargs where the replacement string is used, an argument will be substituted instead. With this approach, only one argument is passed to each command. However, the argument will be created from the entire input string, rather than from a separate fragment of it. You can also use the -L option of the xargs command, which will cause the entire string to be used as an argument, rather than individual pieces separated by spaces. Using the -I option implicitly causes the -L 1 option to be used. Listing 15 shows examples of using the -I and –L options.
Listing 15. Using the xargs command and input lines
$ xargs -I XYZ echo "START XYZ REPEAT XYZ END"Although our examples use simple text files, you won't often use the xargs command for such cases. Typically, you'll be dealing with a large list of files resulting from commands like ls , find or grep . Listing 16 shows one way to pass xargs a list of the contents of a directory to a command such as grep.
Listing 16. Using xargs command and file list
$ ls |xargs grep "1" text1:1 apple text2:10 apple xaa:1 apple yaa:1In the last example, what happens if one or more of the filenames contains spaces? If you try to use the command as in Listing 16, you will get an error. In a real situation, the list of files may not be obtained from the ls command, but, for example, as a result of executing a user script or command; or you may want to process it at other stages in the pipeline for additional filtering. So we're not taking into account the fact that you could simply use grep "1" * instead of the existing logical structure.
In the case of the ls command, you could use the --quoting-style option to force filenames containing spaces to be enclosed in parentheses (or escaped). A better solution (when possible) is to use the -0 option of the xargs command, which causes empty characters (\0) to be used to separate input arguments. Although the ls command does not have an option to use null-terminated filenames as output, many commands can do this.
In Listing 17, we'll first copy file text1 to "text 1" and then give some examples of using a list of filenames containing spaces with the xargs command. These examples help you understand the idea, since it may not be so easy to fully master xargs. In particular, the last example of converting newlines to empty characters would not work if some file names already contained newlines. In the next section of this article, we'll look at a more robust solution, using the find command to generate suitable output that uses null characters as delimiters.
Listing 17. Using the xargs command and files containing spaces in their names
$ cp text1 "text 1" $ ls *1 |xargs grep "1" # error text1:1 apple grep: text: No such file or directory grep: 1: No such file or directory $ ls --quoting-style escape * 1 text1 text\ 1 $ ls --quoting-style shell *1 text1 "text 1" $ ls --quoting-style shell *1 |xargs grep "1" text1:1 apple text 1:1 apple $ # Illustrate -0 option of xargs $ ls *1 | tr "\n" "\0" |xargs -0 grep "1" text1:1 apple text 1:1 appleThe xargs command cannot build arbitrarily long commands. Thus, in Linux, up to kernel version 2.26.3, the maximum command length was limited. If you try to run a command such as rm somepath/* and the directory contains many files with long names, it may fail with an error stating that the argument list is too long. If you're running older versions of Linux or UNIX that may have these limitations, it might be helpful to know how you can use xargs in a way that gets around them.
You can use the --show-limits option to view the default limits for the xargs command, and the -s option to set the maximum length of command output. You can learn about other options from the man pages.
Using the find command with the -exec option or in conjunction with the xargs command
In the " " tutorial, you learned how to use the find command to find files based on their names, modification times, sizes, and other characteristics. Usually, certain actions must be performed on the found files - delete, copy, rename them, and so on. We'll now look at the -exec option of the find command, which works similar to the find command and then passing the output to the xargs command.
Listing 18. Using find with -exec option
$ find text -exec cat text3()\; This is a sentence. This is a sentence. This is a sentence. 1 apple 2 pear 3 banana This is a sentence. This is a sentence. This is a sentence. 9 plum 3 banana 10 appleComparing the results of Listing 18 with what you already know about xargs reveals a few differences.
- You must use () symbols in the command to indicate the substitution location where the file name will be substituted. These characters are not automatically added to the end of the command.
- You must end the command with a semicolon, which must be escaped (\;, ";" or ";").
- The command is executed once for each input file.
Try running find text |xargs cat text3 yourself to see the differences.
Now let's go back to the case where the filename contains spaces. In Listing 19, we tried using the find command with the -exec option instead of the ls and xargs commands.
Listing 19. Using the find command with the -exec option and files containing spaces in their names
$find. -name "*1" -exec grep "1" () \; 1 apple 1 appleSo far so good. However, don't you think that something is missing here? Which files contained the lines found by grep? What's missing here is filenames because find calls grep once for each file, and grep, being a smart command, knows that if it was only given the name of one file, it doesn't need to tell you what it was. .
In this situation, we could use the xargs command, but we already know about the problem with files whose names contain spaces. We also mentioned the fact that the find command can generate a null-delimited list of names thanks to the -print0 option. Modern versions of the find command can be separated not by a semicolon, but by a + sign, so that the maximum possible number of names can be passed in one call to the find command, just like when using xargs . Needless to say, in this case you can only use the () construct once, and that it must be the last parameter of the command. Listing 20 demonstrates both of these methods.
Listing 20. Using find , xargs and files containing spaces in their names
$find. -name "*1" -print0 |xargs -0 grep "1" ./text 1:1 apple ./text1:1 apple $ find . -name "*1" -exec grep "1" () + ./text 1:1 apple ./text1:1 appleBoth of these methods are working and the choice of one of them is often determined only by the personal preferences of the user. Be aware that you may run into problems when piping objects with raw delimiters and whitespace; so if you pass output to xargs , use the -print0 option of find , as well as the -0 option of xargs , which tells you to use null delimiters in the input. Other commands, including tar , also support the -0 option and working with null-delimited input, so you should always use this option for commands that support it, unless you are 100% sure that the input list will not produce you problems.
Our last comment concerns working with a list of files. It's a good idea to always check your list and commands carefully before performing batch operations (such as deleting or renaming multiple files). Having an up-to-date backup when needed can also be invaluable.
Output Splitting
To wrap up this article, we'll take a quick look at one more command. Sometimes you may need to view the output on the screen and save it to a file at the same time. For this you we could redirect the output of a command to a file in one window and then use tail -fn1 to trace the output in another window, but the easiest way is to use the tee command.
The tee command is used in a pipe, and its argument is the name of the file (or names of several files) to which the standard output will be piped. The -a option allows you to not replace the old contents of the file with new content, but to append the data to the end of the file. As discussed when we discussed pipelining, if you want to store both standard output and error stream, you must redirect stderr to stdout before passing data as input to the tee command. Listing 21 shows an example of using the tee command to save output to two files, f1 and f2.
Listing 21. Splitting stdout using the tee command
$ ls text|tee f1 f2 text1 text2 text3 $ cat f1 text1 text2 text3 $ cat f2 text1 text2 text3 Pipe is a unidirectional channel for interprocess communication. The term was coined by Douglas McIlroy for the Unix shell and is named after a pipeline. Pipelines are most often used in shell scripts to link multiple commands by redirecting the output of one command (stdout) to the input (stdin) of the next one, using the pipe symbol '|':cmd1 | cmd2 | .... | cmdN
For example:
$ grep -i “error” ./log | wc -l 43
grep performs a case-insensitive search for the string “error” in the log file, but the result of the search is not printed to the screen, but is redirected to the input (stdin) of the wc command, which in turn performs a line count.
Logics
The pipeline provides asynchronous execution of commands using I/O buffering. Thus, all teams in the pipeline work in parallel, each in its own process.The buffer size since kernel version 2.6.11 is 65536 bytes (64Kb) and is equal to a page of memory in older kernels. When attempting to read from an empty buffer, the read process blocks until data appears. Likewise, if you try to write to a full buffer, the writing process will be blocked until the required space is freed.
It is important that despite the fact that the pipeline operates on file descriptors of I/O streams, all operations are performed in memory, without load on the disk.
All information below is for bash-4.2 shell and 3.10.10 kernel.
Simple debug
The strace utility allows you to trace system calls during program execution:$ strace -f bash -c ‘/bin/echo foo | grep bar' .... getpid() = 13726<– PID основного процесса... pipe() <– системный вызов для создания конвеера.... clone(....) = 13727 <– подпроцесс для первой команды конвеера (echo) ... execve("/bin/echo", ["/bin/echo", "foo"], ..... clone(....) = 13728 <– подпроцесс для второй команды (grep) создается так же основным процессом... stat("/home/aikikode/bin/grep", ... Видно, что для создания конвеера используется системный вызов pipe(), а также, что оба процесса выполняются параллельно в разных потоках.
Lots of bash and kernel source code
Source code, level 1, shell
Since the best documentation is the source code, let's turn to it. Bash uses Yacc to parse input commands and returns 'command_connect()' when it encounters a '|' character.parse.y :
1242 pipeline: pipeline '|' newline_list pipeline 1243 ( $$ = command_connect ($1, $4, '|'); ) 1244 | pipeline BAR_AND newline_list pipeline 1245 ( 1246 /* Make cmd1 |& cmd2 equivalent to cmd1 2>&1 | cmd2 */ 1247 COMMAND *tc; 1248 REDIRECTEE rd, sd; 1249 REDIRECT *r; 1250 1251 tc = $1->type == cm_simple ? (COMMAND *) $1->value.Simple: $1; 1252 rd.dest = 1; 1254 r = make_redirection (sd, r_duplicating_output, rd, 0); ) 1256 ( 1257 register REDIRECT *t; 1258 for (t = tc->redirects; t->next; t = t->next) 1259 ; 1260 t->next = r; 1261 ) 1262 else 1263 tc->redirects = r; 1264 1265 $$ = command_connect ($1, $4, '|'); command 1268 ( $$ = $1; ) 1269 ;
Also here we see the processing of the '|&' character pair, which is equivalent to redirecting both stdout and stderr into the pipeline. Next, let's look at command_connect():make_cmd.c :
194 COMMAND * 195 command_connect (com1, com2, connector) 196 COMMAND *com1, *com2; 197 int connector; 198 ( 199 CONNECTION *temp; 200 201 temp = (CONNECTION *)xmalloc (sizeof (CONNECTION)); 202 temp->connector = connector; 203 temp->first = com1; 204 temp->second = com2; 205 return ( make_command (cm_connection, (SIMPLE_COM *)temp)); where connector is the symbol '|' as int. When executing a sequence of commands (linked via '&', '|', ';', etc.), execute_connection():execute_cmd.c is called:
PIPE_IN and PIPE_OUT are file descriptors containing information about the input and output streams. They can take the value NO_PIPE, which means the I/O is stdin/stdout.
execute_pipeline() is a fairly extensive function, the implementation of which is contained in execute_cmd.c . We will look at the parts that are most interesting to us.
execute_cmd.c :
2112 prev = pipe_in; 2113 cmd = command; 2114 2115 while (cmd && cmd->type == cm_connection && 2116 cmd->value.Connection && cmd->value.Connection->connector == '|') 2117 ( 2118 /* Create a pipeline between two commands */ 2119 if (pipe (fields)< 0) 2120 { /* возвращаем ошибку */ } ....... /* Выполняем первую команду из конвейера, используя в качестве входных данных prev - вывод предыдущей команды, а в качестве выходных fildes - выходной файловый дескриптор, полученный в результате вызова pipe() */ 2178 execute_command_internal (cmd->value.Connection->first, asynchronous, 2179 prev, fildes, fd_bitmap); 2180 2181 if (prev >= 0) 2182 close (prev); 2183 2184 prev = fildes; /* Our output becomes the input for the next command */ 2185 close (fildes); ....... 2190 cmd = cmd->value.Connection->second; /* “Move” to the next command from the pipeline */ 2191 ) So bash processes the pipeline symbol by system call pipe() for each '|' character encountered and executes each command in a separate process using the corresponding file descriptors as input and output streams.
Source code, level 2, kernel
Let's turn to the kernel code and look at the implementation of the pipe() function. This article discusses the kernel version 3.10.10 stable.(code sections that are insignificant for this article are omitted):
/* Maximum size pipeline buffer for an unprivileged user. Can be set by root in the file /proc/sys/fs/pipe-max-size */ 35 unsigned int pipe_max_size = 1048576; The pipeline buffer, according to POSIX recommendations, is equal to the size of one memory page, i.e. 4Kb */ 40 unsigned int pipe_min_size = PAGE_SIZE;
/*
:
Minimum size
The maximum size of a data block that will be written to the pipeline is equal to one memory page (4Kb) for the arm architecture:
fcntl(fd, F_SETPIPE_SZ,
) The maximum allowed buffer size, as we saw above, is specified in the file /proc/sys/fs/pipe-max-size.Tips & tricks
In the examples below we will execute ls on the existing Documents directory and two non-existent files: ./non-existent_file and . /other_non-existent_file.
ls -d ./Documents ./non-existent_file ./other_non-existent_file |& egrep “Doc|other” ls: cannot access ./other_non-existent_file: No such file or directory ./DocumentsRedirect _only_ stderr to pipe
$ ls -d ./Documents ./non-existent_file ./other_non-existent_file 2>&1 >/dev/null | egrep “Doc|other” ls: cannot access ./other_non-existent_file: No such file or directory Shoot yourself in the foot
It is important to maintain order stdout redirects and stderr. For example, the combination '>/dev/null 2>&1' will redirect both stdout and stderr to /dev/null.Obtaining the correct pipeline completion code
By default, the pipeline exit code is the exit code of the last command in the pipeline. For example, take the original command that exits with a non-zero code:
$ ls -d ./non-existent_file 2>/dev/null; echo $? 2 And put it in pipe:
$ ls -d ./non-existent_file 2>/dev/null | wc; echo $?0 0 0 0 Now the pipeline exit code is the wc command exit code, i.e. 0.
Typically, we need to know if an error occurred during the execution of the pipeline. To do this, set the pipefail option, which tells the shell that the pipeline exit code will match the first non-zero exit code of one of the pipeline commands, or zero if all commands completed correctly: Shoot yourself in the foot
$ set -o pipefail $ ls -d ./non-existent_file 2>/dev/null | wc; echo $?
0 0 0 2 You should be aware of “harmless” commands that may return non-zero. This applies not only to working with conveyors. For example, consider the grep example:$ egrep “^foo=+” ./config | awk ‘(print “new_”$0;)’ Here we print all the lines found, adding ‘new_’ at the beginning of each line, or do not print anything if there are no lines
required formatnot found. The problem is that grep fails with code 1 if no matches are found, so if our script had the pipefail option set, this example would fail with code 1:
$ set -o pipefail $ egrep “^foo=+” ./config | awk ‘(print “new_”$0;)’ >/dev/null; echo $? 1 In large scripts with complex structures and long pipelines, this point can be overlooked, which can lead to incorrect results.
Assigning values to variables in a pipeline
$ a=aaa $ b=bbb $ echo “one two” | read a b We now expect the values of variables a and b to be “one” and “two” respectively. In fact, they will remain “aaa” and “bbb”. In general, any change to the values of variables in the pipeline outside of it will leave the variables unchanged:
$ filefound=0 $ find . -type f -size +100k |
while true do read f echo “$f is over 100KB” filefound=1 break # exit after the first file found done $ echo $filefound;- Even if find finds a file larger than 100Kb, the filefound flag will still be set to 0.
There are several possible solutions to this problem:
use set -- $var - This construct will set positional variables according to the contents of the var variable. For example, like in the first example above:
$ var=”one two” $ set -- $var $ a=$1 # “one” $ b=$2 # “two” It must be borne in mind that the script will lose the original positional parameters with which it was called . - transfer all the logic for processing the variable value to the same subprocess in the pipeline:
$ echo “one” | (read a; echo $a;) one
change the logic to avoid assigning variables inside the pipeline. - For example, let's change our find example:
$ filefound=0 $ for f in $(find . -type f -size +100k) # we removed the pipeline, replacing it with a loop do read f echo “$f is over 100KB” filefound=1 break done $ echo $filefound;
(only for bash-4.2 and later) use lastpipe option
- Even if find finds a file larger than 100Kb, the filefound flag will still be set to 0.
