5

In a unix terminal, if I do:

egrep "Stuff" $(ls *.txt)

The result is obvious. But what does

$(ls *.txt) do on its own? I see I can replicate the effect by

egrep "Stuff" *.txt

So what is $(ls *.txt)?

  • 5
    See also Useless use of ls * – David Foerster Aug 10 '17 at 10:03
  • 6
    Also Why not parse ls?. – Ruslan Aug 10 '17 at 10:18
  • You'd only do this if you're writing an obfuscated shell script that supports getting extra grep options from the names of files that end in .txt. Maybe with IFS=$'\n' so you could avoid word-splitting filenames with whitespace other than newline. This is obviously not what anyone wants in normal circumstances; I'm just trying to imagine an actual use-case for what is normally a bad/wrong way to do something. – Peter Cordes Aug 11 '17 at 10:46
  • Fun fact: GNU ls has a --quoting-style=shell-always option which might make this safe (but still worse than just *.txt). Arch Linux builds ls with quoting enabled by default when the output is a terminal, I guess for copy/pasting purposes. – Peter Cordes Aug 11 '17 at 10:50
28

"what does $(ls *.txt) do"

Short answer

$(ls *.txt) gathers a list of file names and then mangles them. Do not use this.

Longer answer

  • ls is intended for human-readable output. As one of ls's maintainers wrote in response to why ls would not offer a --null option:

    If we were to do this then this is the interface we would use. However ls is really a tool for direct consumption by a human, and in that case further processing is less useful. For futher processing, find(1) is more suited. [emphasis added]

    In other words, use of ls for anything other than display to humans is just not supported. ls maintainers, for example, recently changed the default output format to something they though more human-friendly without warning. So, follow their advice: if you are going to do further processing of file names, use find instead of ls.

  • $(...) is command substitution. One consequence of using command substitution is that trailing newline characters are removed. If the last file name in the list happens to contain trailing newlines, they will be removed.

  • Since $(...) is not double-quoted, the text it produces will be subject to:

    • Word splitting

    • Pathname expansion

    The result of both of these is further mangling of the file names.

  • not to mention the issues with file names starting with - because of the missing -- (for both ls and egrep as Ubuntu's egrep being the GNU implementation accepts options even after non-option arguments unless POSIXLY_CORRECT is in the environment).

  • also, if any of those txt files were of type directory, ls would list their content instead of themselves.

  • and if there's no txt files, the output of ls will be empty (though you'll see an error about a missing *.txt file) and as egrep will receive no file argument, it will look for Stuff in its standard input (and seemingly hang).

Example

Let's create 4 files containing Stuff in our directory:

$ echo Stuff | tee file1 file2 'a b c.txt' 'f* .txt'
Stuff
$ ls -Q
"a b c.txt"  "file1"  "file2"  "f* .txt"

Now, let's run the egrep command:

$ egrep "Stuff" $(ls *.txt)
grep: a: No such file or directory
grep: b: No such file or directory
grep: c.txt: No such file or directory
file1:Stuff
file2:Stuff
f* .txt:Stuff
grep: .txt: No such file or directory

Observe that we get 4 error messages about nonexistent files. This is due to word splitting. The result also shows matches with two files, file1 and file2 that should not have been searched because they don't end with .txt. This is because of _pathname expansion`.

The correctly written command produces two successful matches and no errors:

$ egrep -- "Stuff" *.txt
a b c.txt:Stuff
f* .txt:Stuff

Recommended solution

Use:

egrep -- "Stuff" *.txt

or POSIXly:

grep -E -- "Stuff" *.txt

or:

grep -E -e Stuff -- *.txt

This will work with any file name and has none of the limitations of the ls approach.

  • 1
    pretty clear approach – solfish Aug 10 '17 at 7:42
  • 1
    Your first point about alteration of file names is only true when the output goes to a terminal, not when it goes to a pipe like in this case. – Stéphane Chazelas Aug 11 '17 at 6:40
  • @StéphaneChazelas Interesting. I'm seeing that also. Thanks. I just updated the answer to replace that statement with a quote from one of the ls maintainers stating that ls should not be used for anything other than human consumption. – John1024 Aug 17 '17 at 22:41
6

The $( ) runs the command ls *.txt and returns the STDOUT of the command.

What this particular usage is, is a newbie programming mistake on at least three levels:

  1. egrep 'Stuff' *.txt works, like you said, except for files named something like A File.txt
  2. Using ls output for program input is unwise. See reasons
  3. $( ls *.txt) mishandles filenames with spaces and other funny characters. find . -maxdepth 1 -type f -iname '*.txt' -print0 | xargs -0 egrep 'Stuff' is a more bullet-resistant way.

A contrived example of shell glob expansion/confusion, in response to @8bittree is:

$ /bin/ls -l -b
total 36
-rw------- 1 w3 walt 2 Aug 11 13:41 A\ \\012\ File.txt
-rw------- 1 w3 walt 2 Aug 11 13:42 A\ \n\ File.txt
-rw------- 1 w3 walt 2 Aug 11 13:40 A\ File.txt

$ grep "STUFF" $(ls *.txt)
grep: A: No such file or directory
grep: \012: No such file or directory
grep: File.txt: No such file or directory
grep: A: No such file or directory
grep: File.txt: No such file or directory
grep: A: No such file or directory
grep: File.txt: No such file or directory

$ find . -maxdepth 1 -type f -iname '*.txt' -print0 | xargs -0 egrep 'Stuff'

$ find . -maxdepth 1 -type f -iname '*.txt' -print0 | xargs -0 egrep '1'
./A File.txt:1
  • Also, using find's -exec action rather than -print0+xargs -0 would be more efficient, although I'll concede its syntax is bothersome. In this case it would be find [...] -exec egrep 'Stuff' {} \; or find [...] -exec egrep 'Stuff' {} + ; in the second case grep handles multiple files at once, so the result will be prefixed by the file path (which could be disabled with grep's -h flag) – Aaron Aug 10 '17 at 14:17
  • @Aaron I had never seen the "{} +" syntax of find, very interesting. Using "-exec egrep 'Stuff' {} \;" without the + would actually be less efficient since it would fork off a new grep process for every file. Also, I usually run the grep with an additional file of /dev/null so I will get file names listed in the output even if only 1 file ends up in the grep command. e.g. find ... -print0 | xargs -0 egrep 'Stuff' /dev/null – Aner Aug 10 '17 at 21:08
  • @Aner concerning the efficiency of {} \; you're right. Also, grep's -H flag is shorter to type than /dev/null ;) Always good to re-read the command's --help (if not its man) from time to time, such goodies come into existence every few versions. – Aaron Aug 10 '17 at 22:14
  • 1
    find -print0 | xargs is no more robust than grep -E *txt. Globs are absolutely safe with any file name you throw at them. Your find example will also work, yes, but it has no benefit whatsoever over the glob; it's just a lot more complex. – terdon Aug 11 '17 at 9:10
  • 1
    That is not an example of a glob being confused by spaces, that's just a bad example of parsing ls. Using globs means using a glob pattern, not the results of running ls on that glob pattern. As 8bittree pointed out, grep STUFF *txt will work fine, as will for i in *txt; do grep STUFF "$i"; done. There is no file name that will make a glob choke. That's what they're designed for. – terdon Aug 13 '17 at 10:31
3

Wow, for a moment I thought you were wondering about `$(ls *.txt)` (with $(..) plus backquotes)

As the question has already been answered, I will just warn you about one thing. Here is a sequence of commands to show you why it can be dangerous:

  • set -x
    To see what is executed
  • touch 'rm -f *.txt'
    Create a text file named 'rm -f *.txt', I use simple quotes here so Bash won't expand the wildcard and spaces.
  • `$(ls *.txt)`
    Here is the tricky part. The command ls *.txt will be run, then redirect the result to STDOUT. This result, rm -f *.txt will be now executed because of the backquotes.
    So, rm will remove every text files, in our case the file rm -f *.txt

I hope you understand, - as a demonstration just run the previous commands in an empty directory, so you won't break anything.

  • 2
    Since that was added by another user, and since ` ... ` has no effects in question titles, I have removed the superfluous backticks. – muru Aug 10 '17 at 8:18
  • 1
    Yep, you're right, title has changed ! Anyway, I think this won't hurt anyone to understand what double command execution can do. – Sayardiss Aug 10 '17 at 8:54
2
$( ... )

Is a command substitution, it assigns the output of one or more commands as an input to another command. The old syntax is: ` ... `.

The different is that your commands will get run in different shells:

echo $BASHPID $( echo $BASHPID )
5718 7138

It actually invokes a subshell:

echo $BASH_SUBSHELL $( echo $BASH_SUBSHELL )
0 1
0

This question seems to have little utility in the real world but,

A better way to get a usable result is:

echo $(ls *.txt)

Or just

ls *.txt

This will list all files with a .txt extension. To find text files that contain a particular string,

egrep -l "Stuff" *.txt

For more information on search tools, do a

man egrep
  • 2
    How is echo $(ls *.txt) more usable than say echo *.txt. It's definitely a lot less reliable because most of the points listed in John's answer also apply there. Note that ls *.txt doesn't list all files with a .txt extension. You'd need ls -d -- *.txt for that. – Stéphane Chazelas Aug 11 '17 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.