165

I had imagined the simplest way to compare the contents of two similar directories would be something like

diff `ls old` `ls new`

But I see why this doesn't work; diff is being handed a big long list of files on the command line, rather than two streams like I had hoped. How do I pass the two outputs to diff directly?

246

Command substitution `…` substitutes the output of the command into the command line, so diff sees the list of files in both directories as arguments. What you want is for diff to see two file names on its command line, and have the contents of these files be the directory listings. That's what process substitution does.

diff <(ls old) <(ls new)

The arguments to diff will look like /dev/fd/3 and /dev/fd/4: they are file descriptors corresponding to two pipes created by bash. When diff opens these files, it'll be connected to the read side of each pipe. The write side of each pipe is connected to the ls command.

  • 49
    echo <(echo) <(echo) never thought this could be so interesting :D – Aquarius Power Dec 13 '14 at 1:46
  • 3
    Process substitution is not supported by all shells, but pipe redirects are a neat workaround. – Irfan434 Sep 16 '17 at 14:38
  • 1
    Just to mention that parsing ls is not recommended unix.stackexchange.com/questions/128985/why-not-parse-ls – Katu Apr 12 '18 at 7:59
  • @Katu The problem with ls is that it mangles file names. Parsing its output is fragile (it doesn't work with “weird” file names). For comparing two directory listings, it's ok as long as the output is unambiguous. With arbitrary file names, this would require an option such as --quoting-style=escape. – Gilles Apr 12 '18 at 11:03
  • 1
    @will <(…) creates a pipe. It seems that meld doesn't work with pipes, so you can't use <(…). In zsh, you can replace <(…) by =(…) and it will work because =(…) puts the intermediate outputs in a temporary file. In bash I don't think there's any convenient syntax, you'd have to manage the temporary files yourself. – Gilles Aug 28 at 11:33
3

For zsh, using =(command) automatically creates a temporary file and replaces =(command) with the path of the file itself. With Command Substitution, $(command) is replaced with the output of the command.

So there are three options:

  1. Command Substitution: $(...)
  2. Process Substitution: <(...)
  3. zsh-Flavored Process Substitution: =(...)

zsh flavored process subsitution, #3, is very useful and can be used like so to compare the output of two commands using a diff tool, for example Beyond Compare:

bcomp  =(ulimit -Sa | sort) =(ulimit -Ha | sort)

For Beyond Compare, note that you must use bcomp for the above (instead of bcompare) since bcomp launches the comparison and waits for it to complete. If you use bcompare, that launches comparison and immediately exits due to which the temporary files created to store the output of the commands disappear.

Read more here: http://zsh.sourceforge.net/Intro/intro_7.html

Also notice this:

Note that the shell creates a temporary file, and deletes it when the command is finished.

and the following which is the difference between the two types of Process substitution supported by zsh (i.e #2 and #3):

If you read zsh's man page, you may notice that <(...) is another form of process substitution which is similar to =(...). There is an important difference between the two. In the <(...) case, the shell creates a named pipe (FIFO) instead of a file. This is better, since it does not fill up the file system; but it does not work in all cases. In fact, if we had replaced =(...) with <(...) in the examples above, all of them would have stopped working except for fgrep -f <(...). You can not edit a pipe, or open it as a mail folder; fgrep, however, has no problem with reading a list of words from a pipe. You may wonder why diff <(foo) bar doesn't work, since foo | diff - bar works; this is because diff creates a temporary file if it notices that one of its arguments is -, and then copies its standard input to the temporary file.

Reference: https://unix.stackexchange.com/questions/393349/difference-between-subshells-and-process-substitution

  • 2
    $(...) is not process substitution, it's command substitution. <(...) is process substitution. That's why the quoted passage doesn't mention $(...) at all. – muru Jun 29 '18 at 9:10
2

Fish shell

In Fish shell you have to pipe into psub. Here is an example of heroku and dokku config comparison with Beyond Compare:

bcompare (ssh me@myapp.pl dokku config myapp | sort | psub) (heroku config -a myapp | sort | psub)
  • 1
    Another graphical diff tool is meld which is open source and available in the Ubuntu and EPEL repositories. meldmerge.org – phiphi Jun 18 '18 at 7:36
0

I often use the technique described in the accepted answer:

diff <(ls old) <(ls new)

but I find I usually use it with much more complex commands than the example above. In such cases it can be annoying to craft the diff command. I have come up with some solutions that others may find useful.

I find that 99% of the time I try out the relevant commands before running diff. Consequently the commands I want to diff are right there in my history ... why not use them?

I make use of the Fix Command (fc) bash builtin to execute the last two commands:

$ echo A
A
$ echo B
B
$ diff --color <( $(fc -ln -1 -1) ) <( $(fc -ln -2 -2 ) )
1c1
< B
---
> A

The fc flags are:

-n: No number. It suppresses the command numbers when listing.

-l: Listing: The commands are listed on standard output.

the -1 -1 refer to the start and end positing in the history, in this case its from the last command to the last command which yields just the last command.

Lastly we wrap this in $() to execute the command in a subshell.

Obviously this is a bit of a pain to type so we can create an alias:

alias dl='diff --color <( $(fc -ln -1 -1) ) <( $(fc -ln -2 -2 ) )'

Or we can create a function:

dl() {
    if [[ -z "$1" ]]; then
        first="1"
    else
        first="$1"
    fi
    if [[ -z "$2" ]]; then
        last="2"
    else
        last="$2"
    fi
    # shellcheck disable=SC2091
    diff --color <( $(fc -ln "-$first" "-$first") ) <( $(fc -ln "-$last" "-$last") )
}

which supports specifying the history lines to use. After using both I find the alias is the version I prefer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.