In a unix terminal, if I do:
egrep "Stuff" $(ls *.txt)
The result is obvious. But what does
$(ls *.txt)
do on its own? I see I can replicate the effect by
egrep "Stuff" *.txt
So what is $(ls *.txt)
?
"what does $(ls *.txt)
do"
$(ls *.txt)
gathers a list of file names and then mangles them. Do not use this.
ls
is intended for human-readable output. As one of ls's maintainers wrote in response to why ls
would not offer a --null
option:
If we were to do this then this is the interface we would use. However
ls
is really a tool for direct consumption by a human, and in that case further processing is less useful. For futher processing,find
(1) is more suited. [emphasis added]
In other words, use of ls
for anything other than display to humans is just not supported. ls
maintainers, for example, recently changed the default output format to something they though more human-friendly without warning. So, follow their advice: if you are going to do further processing of file names, use find
instead of ls
.
$(...)
is command substitution. One consequence of using command substitution is that trailing newline characters are removed. If the last file name in the list happens to contain trailing newlines, they will be removed.
Since $(...)
is not double-quoted, the text it produces will be subject to:
Word splitting
Pathname expansion
The result of both of these is further mangling of the file names.
not to mention the issues with file names starting with -
because of the missing --
(for both ls
and egrep
as Ubuntu's egrep
being the GNU implementation accepts options even after non-option arguments unless POSIXLY_CORRECT
is in the environment).
also, if any of those txt
files were of type directory, ls
would list their content instead of themselves.
and if there's no txt
files, the output of ls
will be empty (though you'll see an error about a missing *.txt
file) and as egrep
will receive no file argument, it will look for Stuff in its standard input (and seemingly hang).
Let's create 4 files containing Stuff
in our directory:
$ echo Stuff | tee file1 file2 'a b c.txt' 'f* .txt'
Stuff
$ ls -Q
"a b c.txt" "file1" "file2" "f* .txt"
Now, let's run the egrep command:
$ egrep "Stuff" $(ls *.txt)
grep: a: No such file or directory
grep: b: No such file or directory
grep: c.txt: No such file or directory
file1:Stuff
file2:Stuff
f* .txt:Stuff
grep: .txt: No such file or directory
Observe that we get 4 error messages about nonexistent files. This is due to word splitting. The result also shows matches with two files, file1
and file2
that should not have been searched because they don't end with .txt
. This is because of _pathname expansion`.
The correctly written command produces two successful matches and no errors:
$ egrep -- "Stuff" *.txt
a b c.txt:Stuff
f* .txt:Stuff
Use:
egrep -- "Stuff" *.txt
or POSIXly:
grep -E -- "Stuff" *.txt
or:
grep -E -e Stuff -- *.txt
This will work with any file name and has none of the limitations of the ls
approach.
ls
should not be used for anything other than human consumption.
The $( )
runs the command ls *.txt
and returns the STDOUT
of the command.
What this particular usage is, is a newbie programming mistake on at least three levels:
egrep 'Stuff' *.txt
works, like you said, except for files named something like A File.txt
ls
output for program input is unwise. See reasons$( ls *.txt)
mishandles filenames with spaces and other funny characters. find . -maxdepth 1 -type f -iname '*.txt' -print0 | xargs -0 egrep 'Stuff'
is a more bullet-resistant way.A contrived example of shell glob expansion/confusion, in response to @8bittree is:
$ /bin/ls -l -b
total 36
-rw------- 1 w3 walt 2 Aug 11 13:41 A\ \\012\ File.txt
-rw------- 1 w3 walt 2 Aug 11 13:42 A\ \n\ File.txt
-rw------- 1 w3 walt 2 Aug 11 13:40 A\ File.txt
$ grep "STUFF" $(ls *.txt)
grep: A: No such file or directory
grep: \012: No such file or directory
grep: File.txt: No such file or directory
grep: A: No such file or directory
grep: File.txt: No such file or directory
grep: A: No such file or directory
grep: File.txt: No such file or directory
$ find . -maxdepth 1 -type f -iname '*.txt' -print0 | xargs -0 egrep 'Stuff'
$ find . -maxdepth 1 -type f -iname '*.txt' -print0 | xargs -0 egrep '1'
./A File.txt:1
find
's -exec
action rather than -print0
+xargs -0
would be more efficient, although I'll concede its syntax is bothersome. In this case it would be find [...] -exec egrep 'Stuff' {} \;
or find [...] -exec egrep 'Stuff' {} +
; in the second case grep
handles multiple files at once, so the result will be prefixed by the file path (which could be disabled with grep
's -h
flag)
{} \;
you're right. Also, grep
's -H
flag is shorter to type than /dev/null
;) Always good to re-read the command's --help
(if not its man) from time to time, such goodies come into existence every few versions.
find -print0 | xargs
is no more robust than grep -E *txt
. Globs are absolutely safe with any file name you throw at them. Your find
example will also work, yes, but it has no benefit whatsoever over the glob; it's just a lot more complex.
ls
on that glob pattern. As 8bittree pointed out, grep STUFF *txt
will work fine, as will for i in *txt; do grep STUFF "$i"; done
. There is no file name that will make a glob choke. That's what they're designed for.
Wow, for a moment I thought you were wondering about `$(ls *.txt)`
(with $(..)
plus backquotes)
As the question has already been answered, I will just warn you about one thing. Here is a sequence of commands to show you why it can be dangerous:
set -x
touch 'rm -f *.txt'
`$(ls *.txt)`
ls *.txt
will be run, then redirect the result to STDOUT. This result, rm -f *.txt
will be now executed because of the backquotes.rm -f *.txt
I hope you understand, - as a demonstration just run the previous commands in an empty directory, so you won't break anything.
...
` has no effects in question titles, I have removed the superfluous backticks.
$( ... )
Is a command substitution, it assigns the output of one or more commands as an input to another command. The old syntax is: ` ... `
.
The different is that your commands will get run in different shells:
echo $BASHPID $( echo $BASHPID )
5718 7138
It actually invokes a subshell:
echo $BASH_SUBSHELL $( echo $BASH_SUBSHELL )
0 1
This question seems to have little utility in the real world but,
A better way to get a usable result is:
echo $(ls *.txt)
Or just
ls *.txt
This will list all files with a .txt extension. To find text files that contain a particular string,
egrep -l "Stuff" *.txt
For more information on search tools, do a
man egrep
echo $(ls *.txt)
more usable than say echo *.txt
. It's definitely a lot less reliable because most of the points listed in John's answer also apply there. Note that ls *.txt
doesn't list all files with a .txt extension. You'd need ls -d -- *.txt
for that.
Aug 11, 2017 at 10:52
ls *
ls
?.grep
options from the names of files that end in.txt
. Maybe withIFS=$'\n'
so you could avoid word-splitting filenames with whitespace other than newline. This is obviously not what anyone wants in normal circumstances; I'm just trying to imagine an actual use-case for what is normally a bad/wrong way to do something.ls
has a--quoting-style=shell-always
option which might make this safe (but still worse than just*.txt
). Arch Linux buildsls
with quoting enabled by default when the output is a terminal, I guess for copy/pasting purposes.