52

I want merge (union) output from two different commands, and pipe them to a single command.

A silly example:

Commands I want to merge the output:

cat wordlist.txt
ls ~/folder/*

into:

wc -l

In this example, if wordlist.txt contains 5 lines and 3 files, I want wc -l to return 8.

$cat wordlist.txt *[magical union thing]* ls ~/folder/* | wc -l
8

How do I do that?

3 Answers 3

78

Your magical union thing is a semicolon... and curly braces:

    { cat wordlist.txt ; ls ~/folder/* ; } | wc -l

The curly braces are only grouping the commands together, so that the pipe sign | affects the combined output.

You can also use parentheses () around a command group, which would execute the commands in a subshell. This has a subtle set of differences with curly braces, e.g. try the following out:

    cd $HOME/Desktop ; (cd $HOME ; pwd) ; pwd
    cd $HOME/Desktop ; { cd $HOME ; pwd ; } ; pwd

You'll see that all environment variables, including the current working directory, are reset after exiting the parenthesis group, but not after exiting the curly-brace group.

As for the semicolon, alternatives include the && and || signs, which will conditionally execute the second command only if the first is successful or if not, respectively, e.g.

    cd $HOME/project && make
    ls $HOME/project || echo "Directory not found."
9
  • 2
    That does work! Help me learn - what exactly are the curly braces doing in here? What other magical powers do they have? May 7, 2012 at 19:13
  • 1
    @DavidOneill { list; } is a compound command. From bash(1): list is simply executed in the current shell environment. list must be terminated with a newline or semicolon. [..] The return status is the exit status of list.
    – Lekensteyn
    May 7, 2012 at 19:18
  • 1
    I've added a bit more info to the answer. The definitive guide to shell scripting with bash is the bash manpage, type man bash from the command line and browse it.
    – pablomme
    May 7, 2012 at 19:24
  • should the ; in those commands not be && in case the file wordlist.txt does not exists?
    – Rinzwind
    May 7, 2012 at 19:33
  • If wordlist.txt does not exist an error from cat will appear on standard error, but not on standard output, so wc -l will not count any lines from it. Same goes for ~/folder not existing. One could add 2> /dev/null between each of these commands and their semicolon to prevent noise on standard error, but other than being ugly, error messages are harmless for the purpose of counting lines.
    – pablomme
    May 7, 2012 at 19:39
9

Since wc accepts a file path as input, you can also use process substitution:

wc -l <(cat wordlist.txt; ls ~/folder/*)

This roughly equivalent to:

echo wordlist.txt > temp
ls ~/folder/* >> temp
wc -l temp

Mind that ls ~/folder/* also returns the contents of subdirectories if any (due to glob expansion). If you just want to list the contents of ~/folder, just use ls ~/folder.

6
  • 1
    The wc -l was just a made up example, i'm actually piping it into something more complicated that doesn't have this option. May 7, 2012 at 19:12
  • 3
    @DavidOneill Well, as cat accepts a file argument, you can use cat <(cat wordlist.txt; ls wordlist.txt) | wc -l and even cat wordlist.txt <(ls wordlist.txt) | wc -l. This is of course very ugly, but it demonstrates the endless possibilities with command line tools.
    – Lekensteyn
    May 7, 2012 at 19:14
  • 2
    @DavidOneill Correct, I've corrected it now, thanks.
    – Lekensteyn
    May 7, 2012 at 19:25
  • 2
    You want ls ~/folder/ so that if ~/folder is a symbolic link ls lists the contents of its target instead of the link itself. By the way, all the ls commands should be followed by -1 so that a single file per line is printed and wc -l is a valid way of counting files.
    – pablomme
    May 7, 2012 at 20:28
  • @pablomme You're true about the symlink thing, but -1 is implied as the output is not a terminal but a pipe.
    – Lekensteyn
    May 7, 2012 at 20:44
3

I was asking myself the same question, and ended up writing a short script.

magicalUnionThing (I call it append):

#!/bin/sh
cat /dev/stdin
$*

Make that script executable

chmod +x ./magicalUnionThing

Now you do

cat wordlist.txt |./magicalUnionThing ls ~/folder/* | wc -l

What it does:

  • Send standard input to standard output
  • Execute argument. $* returns all arguments as a string. The output of that command goes to script standard output by default.

So the stdout of magicalUnionThing will be its stdin + stdout of the command that is passed as argument.

There are of course simpler ways, as per other answers.
Maybe this alternative can be useful in some cases.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .