Xargs: a variety of uses. Use an escape sequence to enter a long string. Create directory trees with one command

There is a very strange command in Linux xargs , which gurus are very fond of, but are in no hurry to explain how it works. The Internet is littered with recipes on “how to use xargs,” but none of them clearly states the most important thing: what this command actually does.

The most important

IN general outline The same thing is written everywhere: the xargs command takes an input stream (which is why it is always preceded by some command and the stream redirection symbol "|"), and with some magical syntax executes the command specified in it.

This is what the xargs command actually does.(I’ll try to formulate it as impartially as possible). It breaks the stream of symbols sent into it into pieces. It uses delimiter characters to split the stream. And for each selected piece, it executes the command that is indicated on the right side, supplementing this command on the right with symbols of the found piece.

Yes, this definition uses the concept of “right” twice. The details are explained below. In the meantime, it’s better to look at the structure of the xargs command in the form of a picture. Syntactically, the xargs command consists of two parts - left and right:

Moreover, there is an unambiguous visual separation of where is left and where is right part, simply no. If you are trying to understand another person's xargs command, you need to be able to find this "interface" yourself. Here are some examples:

Full team	Left side	Right part	Note
xargs rm -rf	xargs	rm -rf
xargs -0 rm -rf	xargs -0	rm -rf
xargs -p -l gzip	xargs -p -l	gzip
xargs tar -zcf pl.tar.gz	xargs	tar -zcf pl.tar.gz
xargs -n2 fmv	xargs -n2
xargs -I file mv	xargs -I file		Yes, there is no mistake here
xargs chown temp	xargs	chown temp
xargs kill -9	xargs	kill -9
xargs -p vim	xargs -p

That is, the rule applies here: if after xargs there are characters preceded by a minus sign"-" , then these are command options xargs . As soon as there are symbols without a minus sign, then these are already symbols on the right side. But you need to take into account that some options xargs require some other data after themselves that will not be preceded by a minus sign (see example with the option-I ).

And now the most important thing: what command does it execute? xargs ? Where does she push the pack of symbols that she isolated from the input stream? It's simple: she places these symbols to the right of the command written on the right side. I understand that the concept of “right” is used twice here. Then here is a picture that puts everything in its place:

Let's take specific example. The directory contains the following files:

main.cpp

main.h

version.cpp

version.h

config.cpp

config.h

data.cpp

data.h

Inside this directory the command is executed:

$find. -name "*.cpp" | xargs -n 1 rm -rf

What commands will xargs generate? To answer this, you need to understand what will be submitted to its input. And the input will be the result of the find command:

./main.cpp

./version.cpp

./config.cpp

./data.cpp

The xargs command considers the delimiter space, tab, or line feed (and their continuous sequences). Thus, four commands will be executed in the end:

rm -rf ./main.cpp

rm -rf ./version.cpp

rm -rf ./config.cpp

rm -rf ./data.cpp

Very important note about the magic option

There is one very important note. If you don’t understand it, you won’t be able to work properly with xargs, and you’ll become like the authors of articles who think they understand how xargs works, but in fact write terrible nonsense. In the above example, the option is written for a reason"-n 1" .

Option "-n 1" causes xargs to execute the command for each successive chunk from the input stream. Yes, I understand that this sounds crazy: after all, the xargs team is supposed to do exactly that! After all, the manual says the following:"xargs reads items from the standard input, delimited by blanks (which can be protected with double or single quotes or a backslash) or newlines, and executes the command (default is /bin/echo) one or more times with any initial-arguments followed by items read from standard input."The problem is that by default, if you do not specify"-n 1" , xargs treats the entire incoming stream, broken up by spaces, tabs, and line breaks, as ONE argument. And in fact, the entire incoming stream is simply substituted into the command being executed. What a surprise from the developers!

Question: How then do the examples given in articles work, such as

$find. -name "*.cpp" | xargs rm -rf

$find. -name "*.cpp" | xargs wc -l

And they work simply because the commands themselves rm, wc and others like them can work with a set of file names. And users mistakenly think that this is xargs Calls these commands multiple times for each filename. And to make sure of this, you can use the option-t (print the command generated xargs , before executing it). But to see the result, you still need to use the output redirection construct from the error stream 2>&1 (because using the option-t gives output to an error stream rather than to the standard console). And this is what you can see.

If you write the xargs command without the "-n 1" option , then the following will happen:

$find. -name "*.cpp" | xargs -t rm -rf 2>&1

rm -rf ./main.cpp ./version.cpp ./config.cpp ./data.cpp

It can be seen that only one team volunteered rm , and a list of file names is passed to it. It's just that the result of its work will look as if it was called separately for each file.

If you use the option"-n 1" , then the picture will be different:

$find. -name "*.cpp" | xargs-n 1 -t rm -rf 2>&1

rm -rf ./main.cpp

rm -rf ./version.cpp

rm -rf ./config.cpp

rm -rf ./data.cpp

Here the behavior is exactly as promised. Remember this option, and don't be surprised that xargs works somehow wrong if you don't use this option. Also remember that in many articles on the Internet, commands with xargs simply inoperable. The authors think that they know what the result should be, and do not even check the “obvious”, as a result of which an unprepared user who decides to repeat what is written in the article will not really understand anything.

There is one more subtle point. xargs has a limit on the length of the input stream. And if the input stream is too large, xargs will split it into two or more pieces, and for each piece it will still call a separate command indicated on the right side. To avoid such unforeseen situations, use the option"-n 1" .

xargs command without arguments

Sometimes you can come across a discouraging construction such as:

tr -dc A-Za-z0-9_< /dev/urandom | head -c 10 | xargs

This command generates random password 10 characters long. But what does command mean? xargs no arguments at the end of this command?

The answer is simple. Team xargs without arguments, actually thinks that the right side of it is the command/bin/echo . And passes the incoming stream through the command echo . Why is this needed? IN in this example this is simply to ensure that the final result ends with a newline character. Here's an example showing the difference between a team that doesn't have xargs is xargs :

> tr -dc A-Za-z0-9_< /dev/urandom | head -c 10

7jk2qx4cX8>

> tr -dc A-Za-z0-9_< /dev/urandom | head -c 10 | xargs

zSlr2HsbSa

Spaces in file names

Since xargs considers spaces, tabs, and newlines to be delimiters, there is a problem with processing file names containing whitespace characters.

Typically, the file names for the input of the xargs program are supplied from the result of the command. find . And to solve this problem the team find has an option "-print0" . It replaces the line break with a null character\x0 . And the xargs command has an option "-0" (minus zero), with the help of which the input stream is divided into parts separated by the symbol\x0 .

Suppose a file appears in a directory with the name"new file.cpp" . If you do not use the null-to-null conversion options, the following will happen:

$find. -name "*.cpp" | xargs -n 1 -t rm -rf 2>&1

rm -rf ./new

rm -rf file.cpp

and, of course, the file"new file.cpp" will not be deleted. If you add the above options, the command will work correctly:

$find. -name "*.cpp" -print0 | xargs -n 1 -t -0 rm -rf 2>&1

rm -rf ./new file.cpp

and the file will be deleted.

What happens if you don't write the "-n" option?

Please note that the above commands use the option"-n 1" . What will happen if you don’t write it? In principle, everything will work exactly the same. But few people can explain how it works, because visually the commands will be the same, but the result will be different. Here's an example.

Command without "-n 1" option and without null conversion options:

$find. -name "*.cpp" | xargs -t rm -rf 2>&1

"rm..." , and it won't delete the file"new file.cpp" .

And now the command without the "-n 1" option, but with options for converting the null character:

$find. -name "*.cpp" -print0 | xargs -t -0 rm -rf 2>&1

rm -rf ./main.cpp ./data.cpp ./config.cpp ./version.cpp ./new file.cpp

As a result, the team was constructed"rm..." , outwardly absolutely identical to the previous one, but it will delete the file "new file.cpp" !

It's difficult to explain how this works. After all, the option"-0" is the xargs command, not the rm command . In the command man page rm There is no indication that if file names are delimited by null characters, whitespace characters in file names will be treated as literals rather than as delimiters. For the author of the article, this behavior remains a mystery, and so far no specialist has been found who would explain what is actually happening.

The most important question

But how do you construct commands in which you don’t just need to add the found sequences of characters to the right? And if you need to add something else after the value inserted on the right? What should I do? But no way! This is the answer. It is not possible to construct an arbitrary command using xargs. You can only construct a command consisting of a base (fixed) part and a right (wildcard) part. That's all!

If xargs allowed you to add something after the wildcard part, life with this command would be much easier. For example, it would be possible to put quotes before and after the wildcard part, and there would simply be no problem with spaces in file names. But the xargs syntax does not allow for such behavior.

So is it really impossible to construct the required command in *NIX? Of course available. For this you can use awk command and her system function(). How to do this is written in the article: .

I was recently inspired to write this post after I came across situations where I needed to edit several files from a project and replace some words with others. I used to do this through Notepad++(CTRL + H), but this is good if we know in which file we need to replace the word, and if there are thousands of these files. So I want to introduce you to 3 basic tools systems UNIX to date: grep, xargs, And sed.

You may have already heard them before and already know how to use them, but you probably won’t learn anything new. But if you want to learn the basics on using these tools, read on.

Grep

Grep is a base Unix command line for searching phrases from both the standard stream and files. Grep returns information about the files that contain your phrase and a little information about where they are located. Here's an example standard method using Grep:

Grep "footer" index.html

Now let's add some search options:

i- Ignores case when making comparisons.
n- Displays before each line its number in the file.
b- Displays the column number in the found file.

grep -i -n -b "footer" index.html

Now search for whole words:

w- Searches using whole words. For example, when searching for a string normally text grep can find the word textarea. And if used given key then only lines containing the word text will be found.

grep -w "text" index.html

Searching by file is good, but it’s better if we can search for files recursively within the project.

r- Recursive search for files from the current directory.
l- Returns only file names containing the searched value.

grep -r -l "footer" ./

Now let’s try to display files that do not contain the desired value.

L- Displays file names that do not contain the desired value.

grep -r -L -i "footer" ./

Also very often we need to search in specific types files given values.

–include=“PATTERN”- Search in files using a given pattern

grep -r --include="*.html" -l "footer" ./

Search for files that start with index and end with .php

Grep -r --include="index*.php" -l "footer" ./

Now let's transfer the result of the work to a file.

Grep -r -l "footer" ./ > out.txt

Xargs

This amazing command works on a trailer, takes data from standard input or from a file, splits it according to specified parameters and then passes it to another program as an argument. Xargs is executed until the data from the stream is exhausted. Now let's look at some simple examples.

Break the standard stream into arguments.

–n- Specify the number of returned arguments.

echo 1 2 3 4 | xargs -n 1

Now let's set the argument separator.

–d- Points to the separator for outputting arguments.

echo "1-2-3-4" | xargs -d "-" -n 1

Here's more interesting example where xargs is used to transfer files that need to be deleted.

Find. -name "*.txt"| xargs rm -rf

The whole point using xargs This is the splitting of incoming arguments for subsequent transmission of arguments to other programs.

Sed

Sed is a stream editor that takes text either from the standard stream or from a text file, performs some operations on the strings and then outputs the result to standard output or to a file. Typically in scripts, sed is used in a data pipeline, in conjunction with other commands and utilities.

Let's do simple replacement lines from the text.txt file and save it to the out.txt file.

Sed "s/text/test/g" text.txt > out.txt

Now let's save all the changes to the current file.

i- to write changes to the current file.

sed -i "s/text/test/g" text.txt

Let's combine everything together

By combining these three commands that are common to all unix systems, we can make a pretty powerful text replacement script. Most of the action comes from using the sed command, but other commands are auxiliary to constructing the query. Now let's look at an example of using all three sweep programs:

Grep -r -l "foo" ./ | xargs sed -i "s/foo/bar/g"

This way we were able to replace everything foo on bar using simple combination from these programs.

Now that we've learned the basics of great tools like grep, xargs, and sed. We will be able to write more advanced scripts for our tasks.

A lot has been written about the xargs utility - what more can be written? But if, as they say, you dig deeper, it turns out that many publications outline only the very basics, but lack the main thing: they do not explain how xargs can be used in real practice. Articles analyzing complex and non-trivial applications of this very useful system administrator Unfortunately, there are very few tools. That is why we wrote our article and tried to include in it as many examples of using xargs to solve various problems.

First, we will look at how xargs works and analyze simpler examples, and then move on to analyzing complex and interesting cases.

Let's remember the basics

The way xargs works can be described as follows: a program takes data from standard input or from a file, splits it according to specified parameters, and then passes it to another program as an argument.

IN general view The xargs command syntax can be represented as follows:

[list_generator_command] | xargs [xargs_options] [command]
Let's look at how it all works using simple and textbook examples.

Deleting files

One of the most common situations in which xargs is used is to delete files found using the find command.

Let's imagine the following situation: there is a directory in which a large number of files. You need to remove files of a certain type from it (in our example, files with the *.sh extension). To perform this operation, you need to pass the output of the find command to xargs, and the -rm command will be applied to files with the specified extension:

$ ls one.sh one.py two.sh two.py $ find . -name "*.sh"| xargs rm -rf $ ls one.py two.py

Note that the operation of deleting files can be carried out without xargs, but using the command

$find. -name "*.sh" -exec rm -rf "()" \

The described method will not work if the name of one of the files to be deleted contains a space. A name consisting of two words separated by a space will not be accepted as a single whole.

Let's illustrate this with the following example:

$ ls new file.sh one.sh one.py two.sh two.py $ find . -name "*.sh"| xargs rm -rf $ ls new file.sh one.py two.py

As you can see, the file with a space in its name was not deleted.

To solve this problem, use the print0 option for the find command and the -0 option for the xargs command. It replaces the standard delimiter (line break with a null character (\x0), which means the end of the stored string:

$find. -name "*.sh" -print0 | xargs -0 rm -rf

Xargs can also help, for example, quickly remove everything temporary files having tmp extension:

$ find /tmp -name "*.tmp"| xargs rm

File compression

You can compress all files in the current directory using gzip by entering the following command:

$ls | xargs -p -l gzip

Let's look at another example: compressing all files with a *.pl extension using tar:

$find. -name "*.pl" | xargs tar -zcf pl.tar.gz

Renaming files

With xargs you can do mass renaming files. Let's imagine that we have a group of files with the extension *.txt, and we need to replace this extension with *.sql. This can be done using xargs and streaming text editor sed:

$ls | sed -e "p;s/.txt$/.sql/" | xargs -n2 fmv

As a result of its execution, a list of renamed files will be displayed on the console.

With xargs you can also add to additional elements to file names (for example, date):

$ls | xargs -I FILE mv()<...>-{}

Instead of<..>you can substitute anything you want.
The curly braces () in this example mean the "current argument" (i.e. the current file name).

Changing permissions for folders and files

Using xargs you can also speed up the process of changing permissions on files and folders for specific user or groups. Let's say we need to find all folders user root and replace their owner with temp. This operation is carried out using the command:

$find. -group root -print | xargs chown temp

To find all folders of the root group and replace the group with temp, use the command:

$find. -group root -print | xargs chgrp temp

Xargs and find: complex operations

You can perform more complex operations using the find and xargs commands. This is how, for example, you can delete temporary files created more than 7 days ago:

$ find /tmp -type f -name "*" -mtime +7 -print0 | xargs -0 rm -f

And here’s how to forcefully stop processes that have already been running for more than 7 days:

$ find /proc -user myuser -maxdepth 1 -type d -mtime +7 -exec basename() \; | xargs kill -9

Xargs and cut

Xargs is quite often used in combination with the cut command, which allows you to cut lines from text files. Let's look at some practical examples. Using the command below, a list of all users on the system will be displayed on the console:

$ cut -d: -f1< /etc/passwd | sort | xargs echo

And the team looks

File * | grep ASCII | cut -d":" -f1 | xargs -p vim
will sequentially open files for editing in vim.
Let's pay attention to the -p option. Thanks to it, the command will be executed in interactive mode: confirmation (y/n) will be requested before opening each file.

In conclusion, we give another complex and interesting example - a recursive search for files of the big size in some directory:

$find. -type f -printf "%20s %p\n" | sort -n | cut -b22- | tr "\n" "\000" | xargs -0 ls -laSr

Parallel running of processes

Xargs is often used for parallel launch several processes. This is how, for example, you can simultaneously compress several directories into tar.gz:

$ echo dir1 dir2 dir3 | xargs -P 3 -I NAME tar czf NAME.tar.gz NAME

The example below uses the -P switch. He points maximum amount processes that will run simultaneously. Let's assume that we have 10 arguments as input. If we enter the xargs command with the -P 3 switch, then 3 instances of the command following xargs will be launched, with each of these arguments.

With xargs you can also download multiple files from the Internet in parallel:

In the given example with specified address all will be downloaded graphic files With jpg extension; The -P switch indicates that you need to download 10 files at a time.

Preliminary results

Let's summarize the preliminary results and formulate a few rules for working with xargs.

Xargs does not work with files that have a space in their name. To solve this problem with the xargs command, use the −0 option. You can also bypass the space in the file name as follows:
$ xargs -I FILE my_command “FILE”
The xargs command accepts commands from standard input, separated by a space or newline. To group these commands, you can use double or single quotes. You can also specify the delimiter using the -d option;
If no arguments are passed to xargs, the command /bin/echo will be executed by default;
In many cases, the xargs command can be replaced with a for loop. For example, the command
$find. -type f -and -iname "*.deb" | xargs -n 1 dpkg -I
completely equivalent to the cycle
$ for file in `find . -type f -and -iname "*.deb"`; do dpkg -I "$file"; done

Non-trivial examples

We've remembered the basics, we've looked at typical use cases... Let's now move on to more complex and non-trivial examples. We came up with some of them ourselves while working on everyday tasks, and some were taken from the site http://www.commandlinefu.com (we highly recommend that anyone who wants to learn the intricacies of working with the command line visit it from time to time - there you can sometimes find very useful tips).

Ban IP addresses from the list

To ban IP addresses from the list, you need to add them to IP tables with the DROP rule. This operation is carried out using the command:

$ cat bad_ip_list | xargs -I IP iptables -A INPUT -s IP -j DROP
You can also perform a more complex operation and ban all addresses by AS:

$ /usr/bin/whois -H -h whois.ripe.net -T route -i origin AS<номер>|egrep "^route"|awk "(print $2)" |xargs -I NET iptables -A INPUT -s NET -j DROP

Changing the URL format

You can convert a URL like “http%3A%2F%2Fwww.google.com” to “www ,google.com” using the command:

Echo "http%3A%2F%2Fwww.google.com" | sed -e"s/%/\\\\\x\1/g" | xargs echo -e

Generate a 10-character password

Generate strong password can be done using a command like:

$ tr -dc A-Za-z0-9_< /dev/urandom | head -c 10 | xargs

You can generate passwords without the help of xargs: there is a specialized utility for this, pwgen. Some other methods for generating passwords are also described.

Looking for binaries installed without using dpkg

Such an operation may be required if, for example, the car has become a victim hacker attack and malware was installed on it software. It will help to identify what programs the attackers installed. next command(it looks for running "binaries" installed without using the dpkg package manager):

$ cat /var/lib/dpkg/info/*.list > /tmp/listin ; ls /proc/*/exe |xargs -l readlink | grep -xvFf /tmp/listin; rm /tmp/listin

Removing outdated kernel packages

The problem of removing old kernels has already been discussed on Habré - see (at the same link you can find interesting examples of commands).

Converting the script to a string

Sometimes there is a need to convert a large script into one line. You can do it like this: Add tags

A lot has been written about the xargs utility - what more can be written? But if, as they say, you dig deeper, it turns out that many publications outline only the very basics, but lack the main thing: they do not explain how xargs can be used in real practice. Unfortunately, there are very few articles analyzing complex and non-trivial options for using this tool, which is very useful for a system administrator. That is why we wrote our article and tried to include as many examples of using xargs to solve various problems as possible.

First, we will look at how xargs works and analyze simpler examples, and then move on to analyzing complex and interesting cases.

Let's remember the basics

In general, the syntax of the xargs command can be represented as follows:

[list_generator_command] | xargs [xargs_options] [command]

Let's look at how it all works using simple and textbook examples.

Deleting files

One of the most common situations in which xargs is used is to delete files found using the find command.

Let's imagine the following situation: there is a directory in which a large number of files are stored. You need to remove files of a certain type from it (in our example, files with the *.sh extension). To perform this operation, you need to pass the output of the find command to xargs, and the -rm command will be applied to files with the specified extension:

$ ls one.sh one.py two.sh two.py $ find . -name "*.sh"| xargs rm -rf $ ls one.py two.py

Note that the operation of deleting files can be carried out without xargs, but using the command

$find. -name "*.sh" -exec rm -rf "()" \

The described method will not work if the name of one of the files to be deleted contains a space. A name consisting of two words separated by a space will not be accepted as a single whole.

Let's illustrate this with the following example:

$ ls new file.sh one.sh one.py two.sh two.py $ find . -name "*.sh"| xargs rm -rf $ ls new file.sh one.py two.py

As you can see, the file with a space in its name was not deleted.

$find. -name "*.sh" -print0 | xargs -0 rm -rf

Xargs can also help, for example, quickly delete all temporary files with a tmp extension:

$ find /tmp -name "*.tmp"| xargs rm

File compression

You can compress all files in the current directory using gzip by entering the following command:

$ls | xargs -p -l gzip

Let's look at another example: compressing all files with a *.pl extension using tar:

$find. -name "*.pl" | xargs tar -zcf pl.tar.gz

Renaming files

Using xargs you can mass rename files. Let's imagine that we have a group of files with the extension *.txt, and we need to replace this extension with *.sql. This can be done using xargs and the streaming text editor sed:

$ls | sed -e "p;s/.txt$/.sql/" | xargs -n2 fmv

As a result of its execution, a list of renamed files will be displayed on the console.

You can also use xargs to add additional elements to filenames (such as a date):

$ls | xargs -I FILE mv()<...>-{}

Instead of<..>you can substitute anything you want.
The curly braces () in this example mean the "current argument" (i.e. the current file name).

Changing permissions for folders and files

Using xargs, you can also speed up the process of changing permissions on files and folders for a specific user or group. Let's say we need to find all root user folders and change their owner to temp. This operation is carried out using the command:

$find. -group root -print | xargs chown temp

To find all folders of the root group and replace the group with temp, use the command:

$find. -group root -print | xargs chgrp temp

Xargs and find: complex operations

You can perform more complex operations using the find and xargs commands. This is how, for example, you can delete temporary files created more than 7 days ago:

$ find /tmp -type f -name "*" -mtime +7 -print0 | xargs -0 rm -f

And here’s how to forcefully stop processes that have already been running for more than 7 days:

$ find /proc -user myuser -maxdepth 1 -type d -mtime +7 -exec basename() \; | xargs kill -9

Xargs and cut

$ cut -d: -f1< /etc/passwd | sort | xargs echo

And the team looks

File * | grep ASCII | cut -d":" -f1 | xargs -p vim

will sequentially open files for editing in vim.
Let's pay attention to the -p option. Thanks to it, the command will be executed in interactive mode: confirmation (y/n) will be requested before opening each file.

In conclusion, here is another complex and interesting example - a recursive search for the largest files in a certain directory:

$find. -type f -printf "%20s %p\n" | sort -n | cut -b22- | tr "\n" "\000" | xargs -0 ls -laSr

Parallel running of processes

Xargs is often used to run multiple processes in parallel. This is how, for example, you can simultaneously compress several directories into tar.gz:

$ echo dir1 dir2 dir3 | xargs -P 3 -I NAME tar czf NAME.tar.gz NAME

The example below uses the -P switch. It specifies the maximum number of processes that will run simultaneously. Let's assume that we have 10 arguments as input. If we enter the xargs command with the -P 3 switch, then 3 instances of the command following xargs will be launched, with each of these arguments.

With xargs you can also download multiple files from the Internet in parallel:

In the example given, all graphic files with the jpg extension will be downloaded from the specified address; The -P switch indicates that you need to download 10 files at a time.

Preliminary results

Let's summarize the preliminary results and formulate a few rules for working with xargs.

Xargs does not work with files that have a space in their name. To solve this problem with the xargs command, use the −0 option. You can also bypass the space in the file name as follows: $ xargs -I FILE my_command “FILE”
The xargs command accepts commands from standard input, separated by a space or newline. To group these commands, you can use double or single quotes. You can also specify the delimiter using the -d option;
If no arguments are passed to xargs, the command /bin/echo will be executed by default;
In many cases, the xargs command can be replaced with a for loop. For example, the $find command. -type f -and -iname "*.deb" | xargs -n 1 dpkg -I
completely equivalent to the cycle
$ for file in `find . -type f -and -iname "*.deb"`; do dpkg -I "$file"; done

Non-trivial examples

We've remembered the basics, we've looked at typical use cases... Let's now move on to more complex and non-trivial examples. We came up with some of them on our own while working on everyday tasks, and some we gleaned from the site http://www.commandlinefu.com (we highly recommend that everyone who wants to learn the intricacies of working with the command line visit it from time to time - there you can sometimes find very useful tips).

Ban IP addresses from the list

To ban IP addresses from the list, you need to add them to IP tables with the DROP rule. This operation is carried out using the command:

$ cat bad_ip_list | xargs -I IP iptables -A INPUT -s IP -j DROP

You can also perform a more complex operation and ban all addresses by AS:

$ /usr/bin/whois -H -h whois.ripe.net -T route -i origin AS<номер>|egrep "^route"|awk "(print $2)" |xargs -I NET iptables -A INPUT -s NET -j DROP

Changing the URL format

You can convert a URL like “http%3A%2F%2Fwww.google.com” to “http://www,google.com” using the command:

Echo "http%3A%2F%2Fwww.google.com" | sed -e"s/%/\\\\\x\1/g" | xargs echo -e

Generate a 10-character password

You can generate a strong password using a command like:

$ tr -dc A-Za-z0-9_< /dev/urandom | head -c 10 | xargs

You can generate passwords without the help of xargs: there is a specialized utility for this, pwgen. Some other methods for generating passwords are also described.

Looking for binaries installed without using dpkg

Such an operation may be required if, for example, the machine was the victim of a hacker attack and malicious software was installed on it. The following command will help you identify what programs the attackers installed (it looks for running “binaries” installed without using the dpkg package manager):

$ cat /var/lib/dpkg/info/*.list > /tmp/listin ; ls /proc/*/exe |xargs -l readlink | grep -xvFf /tmp/listin; rm /tmp/listin

Removing outdated kernel packages

The problem of removing old kernels has already been discussed on Habré - see (at the same link you can find interesting examples of commands).

Converting the script to a string

Sometimes there is a need to convert a large script into one line. You can do it like this:

$ (sed "s/#.*//g"|sed "/^ *$/d"|tr "\n" ";"|xargs echo)< script.sh

Conclusion

As can be seen from the review, the capabilities of xargs are much wider than it might seem at first glance. We hope that the examples given in this article will be useful to you. If you know others interesting options using xargs - welcome to comments.

On long winter evenings I sat and thought “he’ll come daylight hours, I’ll sit down and properly figure out this mysterious xarg utility." Well, it seemed like the time had come - I sat down to figure it out. The first thing that caught my eye was that the man for it was quite mysterious, and didn’t clear things up the first time. Wikipedia also didn’t add any insight, but rather even confused me, so I decided to conduct my own investigation and write a small manual about this. As you know, as long as you explain it, you’ll understand it yourself :)

So, xargs.

xargs is such a utility command line, which allows you to call any command with arguments taken from standard input. Moreover, the arguments can be passed all at once, or they can be grouped into several pieces. We will study xargs version 4.4.0, and according to the man's recommendation, we will use only new arguments that are not marked as deprecated (it is better to immediately get used to working correctly).

So the first thing to understand is how xargs processes the incoming stream and divides it into arguments. There are several modes depending on the options:

1. Normal. By default, the argument separator is any whitespace character: space, tab, vertical tab, or newline. But as in command shell you can use "" or \ to prevent argument splitting.

2. Regular, with grouping. Mode enabled by the -L parameter. Almost identical to the previous one, except that xargs remembers which argument is on which line. Moreover, if the line ends with a space or tab, next line is considered a continuation of the current one.

3. Line by line. Enabled when using the -I or -0 option. In this case, the entire string is considered one whole argument, despite spaces and tabs inside. For -I the end of the line is the character "\n" and for -0 the character "\0"

Let's do a couple of tests to better understand all this. Let's create a file test with the following contents (== no need to enter into the file):
==
arg1
arg2 space
"arg3 quoted"
arg4\ escaped
arg5 with
continue
==
(There must be a space after "arg5 with")
And we’ll also write small script tp, which will output its arguments separated by the symbol ":" and the number:
==
#!/bin/bash
echo -n "@$#"
while [[ $1 != "" ]]; do echo -n ":$1"; shift; done
echo
==

Normal mode (selecting arguments by whitespace characters):
x $ cat test | xargs ./tp
@8:arg1:arg2:space:arg3 quoted:arg4 escaped:arg5:with:continue
The file was split into arguments using whitespace characters, but the lines quoted and escaped with the "\" character remained intact.

The normal mode with grouping by rows is no different from the previous one at this stage.

Line splitting. Let's create a second test file with the following command:
x $ cp test testz && printf "\0arg6" >> testz
Let's check
x $ cat testz | xargs -0 ./tp
@2:arg1
arg2 space
"arg3 quoted"
arg4\ escaped
arg5 with
continue
:arg6

As you can see, there are only 2 arguments. The first is long, preserving newlines, quotes and \, and the second is arg6. In the file they are separated by a null character.

Regarding the separation of parameters, one more thing can be said about the -d option, which specifies a new separator. For example, let's try using "3" as a separator.
x $ cat test | xargs -d 3 ./tp
@2:arg1
arg2 space
"arg: quoted"
arg4\ escaped
arg5 with
continue
The file has been divided into 2 parts at the location of the "3" symbol. What’s remarkable is that this way you can emulate the -0 option
x $ cat testz | xargs -d "\x00" ./tp
@2:arg1
arg2 space
"arg3 quoted"
arg4\ escaped
arg5 with
continue
:arg6

It seems that we have sorted out the division of the input stream into arguments, let's move on to parameter substitution.

So, after its options, xarg waits for a command, which it will execute. All incoming arguments are divided into groups, after which given command is called for each group, and all arguments from that group are passed to it.

Now let's look at how groups are formed.

1. If there are no options, then there is only one group, all arguments from the input stream fall into it. A group of infinite size, so to speak :)

2. The -L n option specifies grouping by lines. The command is passed arguments on n lines. I will demonstrate with examples.
Group by 1 line:
x $ cat test | xargs -L 1 ./tp
@1:arg1
@2:arg2:space
@1:arg3 quoted
@1:arg4 escaped
@3:arg5:with:continue
You can see that the second line contains 2 arguments, because they are both on the same line. And the last one is actually 3, since the penultimate line is “lengthened” due to the space at the end.

Now group by 2 lines. The command contains lines 1 and 2; 3 and 4; and orphan 5th:
x $ cat test | xargs -L 2 ./tp
@3:arg1:arg2:space
@2:arg3 quoted:arg4 escaped
@3:arg5:with:continue

3. Grouping by arguments, specified by the -n x option. Everything is transparent here: the arguments are grouped into x pieces and passed to the command.
For one argument:
x $ cat test | xargs -n 1 ./tp
@1:arg1
@1:arg2
@1:space
@1:arg3 quoted
@1:arg4 escaped
@1:arg5
@1:with
@1:continue
2 arguments each:
x $ cat test | xargs -n 2 ./tp
@2:arg1:arg2
@2:space:arg3 quoted
@2:arg4 escaped:arg5
@2:with:continue

3. Substitution mode - option -I. To begin with, we must recall that in this mode Arguments from the input stream are parsed differently. Each line is one whole argument; lines are not concatenated. Secondly, the -I option has a parameter - a string, which is replaced in the command with an argument:
x $ echo -e "A B\nC D" | xargs -I _ ./tp =_+_=
@1:=A B+A B=
@1:=C D+C D=
It is easy to notice that the _ character is specified as an argument substitution string, which is used 2 times in the command. You can also see that the arguments are allocated as entire lines, and the space does not affect the parsing. The command is called for each argument.

That's all with substitution. Let's look at the remaining important options
-r - do not execute the command if there are no arguments:
x $ cat /dev/null | xargs ./tp
@0
x $ cat /dev/null | xargs -r ./tp
x$
As you can see, in the second case the command was not executed.

P - xargs will ask for confirmation to execute each command.

This completes the short tutorial. It turned out to be not very short, but I hope it was understandable;)