0

I'm new to linux systems and I can't really understand why wee need two operators that can redirect output: pipe as | and ouput redirection operator >? Can't we just always use the second? Most of the times I see that the pipe is used if multiple commands are chained together. If however, the output is redirected to file, as in echo 'hello' > filename, the output redirection operator is used. What am I missing here?

  • What its input redirection? – Avinash Raj Jun 26 '14 at 10:34
2

The key point to remember is that pipes are inter-process communication device that allows two processes ( and that's what commands really are) to exchange data, while redirection operators are for manipulating where particular process writes.

In the video Unix Pipeline, the creator of awk language and one of the original people who worked on AT&T Unix Brian Kernighan explains:

First, you don't have to write one big massive program - you've got existing smaller programs that may already do parts of the job...Another is that it's possible that the amount of data you're procesing would not fit if you stored it in a file...because remember, we're back in the days when disks on these things had, if you were lucky, a Megabyte or two of data...So the pipeline never had to instantiate the whole output

As you can see, within the context which the pipelines were created, they actually were not just communication device, but also save storage space and simplify the development. Sure, we can use output/input redirection for everything (especially nowadays with storage capacity being in the range of Terabytes), however that would be inefficient from the storage point of view, and also processing speed - remember that you're directly feeding output from one command to another with |. Consider something like command1 | grep 'something'. If you write output of command1 first to a file, it will take time to write everything, then let grep go through the whole file. With pipeline and the fact that the output is buffered (meaning that left-side process pauses before right-side process is ready to read again), the output goes directly from one command to the other, saving time.

It is worth noting, that for inter-process communication, there's a use case of named pipes, to which you can use > operator to write from one command, and < to let another command read from it, and it's a use case where you do want to have particular destination on filesystem where multiple scripts/commands can write to and agree on that particular destination. But when it's unnecessary, anonymous pipe | is all you really need.

  • I got to use named pipes for progress bar display and yad. It was interesting the way the file is of FIFO (First In First Out) format and after you read a line it automatically disappears from the file. – WinEunuuchs2Unix Aug 20 '18 at 2:30
  • @WinEunuuchs2Unix Yep, these things have a lot of potential. And remember - they only have name on the filesystem, but otherwise serve as conduit for data and therefore won't store anything ( well, technically the kernel can store small buffered amount while the named pipe is waiting to be read, but nothing actually goes to disk - all in memory buffers). – Sergiy Kolodyazhnyy Aug 20 '18 at 2:35
  • In most cases the progress bar is only 0 to 100 so it's relatively safe having a latency between main script continuously writing to the pipe on integer change and separate sleeping progress bar script that wakes up 4 to 10 times per second reading from the pipe. But it is good you reminded me about the cache limits in the case of a program bug and the writing script overflowing the buffer. – WinEunuuchs2Unix Aug 20 '18 at 2:42
  • Thanks a lot for you elaborate and in-depth answer, exactly what I was looking for back then. Are you on Twitter? – Max Koretskyi aka Wizard Aug 23 '18 at 5:40
  • @AngularInDepth I am on Twitter but don't really post anything – Sergiy Kolodyazhnyy Aug 23 '18 at 5:45
4

I believe the < > operators are used for reading/writing files whereas the | symbol is used for piping the stdout of one command to another.

cal | less 

lets you view the output of cal in a command called less.

cal > less

puts the output of cal into a file called less.

  • aha, so output redirection - >- is used only to write to files? I can't chain commands using output redirection? – Max Koretskyi aka Wizard Jun 26 '14 at 10:40
  • @Maximus you can't do that. – Avinash Raj Jun 26 '14 at 10:40
4
  • | are used to send the output of one command as input to the another command which comes next after the pipe symbol.

    $ echo foo | grep -o 'f'
    f
    
  • To redirect the output of one command to a file , you may use an output redirection > operator.

    $ echo foo > file1
    

    It writes foo to file1. You don't need to manually create that file.

  • If you want to redirect the output to many files then you have to use tee command.

    echo foo | tee file1 file2
    

    It writes foo to file1 and file2. You don't need to manually create that files. Now the file1 and file2 contains only the string foo.

  • Extra thanks for mentioning writing to multiple files with tee – Max Koretskyi aka Wizard Jun 26 '14 at 10:40
  • Note that output redirection doesn't just allow you to write to files but also to things in /dev or /sys, for example, and also allows redirection from one output stream to another (eg STDERR to STDOUT, etc). – thomasrutter Aug 20 '18 at 1:55
3

There's a lot of talk about output redirection but I thought this question was about input. I'm going to ignore > and >> because they've nothing to do with input. Instead I'm going to focus on <, <(...) and |:

  • < expects to read from a file into STDIN while,
  • <(...) provides a file handle to the STDOUT of a command (... here)
  • | pipes STDOUT from one process into the STDIN of the next

So the < isn't directly equivalent to a pipe (it's reading from a file) and the <(...) is reading from the right place, but it's giving a file handle as an output. You need to combine them to offer an equivalent to a pipe.

a | b
< <(a) b

Just reading that, I hope that full explains why the pipe exists. It's much more readable.

  • Thanks a lot for your answer! Could you please explain this line < <(a) b - I didn't really understand it. Appreciate – Max Koretskyi aka Wizard Jun 26 '14 at 16:48
  • 1
    Sure. <(a) is taking the output of a and making it available as a file. The other < is reading that file and turning that into STDIN for b. – Oli Jun 26 '14 at 17:04
  • Got you, thanks! Why didn't you write that way: b < <(a), or than it's going to do different thing? And what is a and b in your examples? Is a a file and b a command? – Max Koretskyi aka Wizard Jun 27 '14 at 5:11
  • 1
    That's exactly the same; I just left it that way around for easier comparison with the pipe. They're both commands just as they would be in a | b – Oli Jun 27 '14 at 7:25
  • Worth noting, that underneath the hood the file offered by <(a) is actually a pipe. See Why does process substitution result in a file called /dev/fd/63 which is a pipe? – Sergiy Kolodyazhnyy Aug 20 '18 at 1:46

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.