31

I was trying to redirect both stdout and stderr to a file today, and I came across this:

<command> > file.txt 2>&1

This apparently redirects stderr to stdout first, and then the resulting stdout is redirected to file.txt.

However, why isn't the order <command> 2>&1 > file.txt? One would naturally read this as (assuming execution from left to right) the command being executed first, the stderr being redirected to stdout and then, the resulting stdout being written to file.txt. But the above only redirects stderr to the screen.

How does the shell interpret both the commands?

  • 7
    TLCL says this "First we redirect standard output to the file, and then we redirect file descriptor 2 (standard error) to file descriptor one (standard output)" and "Notice that the order of the redirections is significant. The redirection of standard error must always occur after redirecting standard output or it doesn't work" – Zanna Dec 13 '16 at 8:37
  • @Zanna: Yes, my question precisely came from reading that in TLCL! :) I'm interested in knowing why it wouldn't work, i.e., how the shell interprets commands in general. – Train Heartnet Dec 13 '16 at 9:19
  • Well, in your question you say the opposite "This apparently redirects stderr to stdout first..." - what I understand from TLCL, is that the shell sends stdout to the file and then stderr to stdout (ie to the file). My interpretation is that if you send stderr to stdout, then it will be displayed in the terminal, and subsequent redirection of stdout will not include stderr (ie the redirection of stderr to the screen completes before the redirection of stdout happens?) – Zanna Dec 13 '16 at 9:23
  • 7
    I know this is an old-fashioned thing to say, but your shell comes with a manual that explains these things - e.g. redirection in the bash manual. By the way, redirections are not commands. – Reinier Post Dec 13 '16 at 9:30
  • The command can't be executed before the redirections are set up: When the execv-family syscall is invoked to actually hand the subprocess off to the command being started, the shell is then out of the loop (no longer has code executing in that process) and has no possible way to control what happens from that point on; redirections thus must all be performed before execution begins, while the shell has a copy of itself running in the process it fork()ed to run your command in. – Charles Duffy Dec 14 '16 at 17:12
41

When you run <command> 2>&1 > file.txt stderr is redirected by 2>&1 to where stdout currently goes, your terminal. After that, stdout is redirected to the file by >, but stderr is not redirected with it, so stays as terminal output.

With <command> > file.txt 2>&1 stdout is first redirected to the file by >, then 2>&1 redirects stderr to where stdout is going, which is the file.

It may seem counter intuitive to start with, but when you think of the redirections in this way, and remember that they are processed from left to right it makes much more sense.

  • It makes sense if you think in terms of file-descriptors and "dup/fdreopen" calls being executed in order from left-to-right – Mark K Cowan Dec 14 '16 at 13:44
20

It might make sense if you trace it out.

In the beginning, stderr and stdout go to the same thing (usually the terminal, which I call here pts):

fd/0 -> pts
fd/1 -> pts
fd/2 -> pts

I'm referring to stdin, stdout and stderr by their file descriptor numbers here: they are file descriptors 0, 1 and 2 respectively.

Now, in the first set of redirections, we have > file.txt and 2>&1.

So:

  1. > file.txt: fd/1 now goes to file.txt. With >, 1 is the implied file descriptor when nothing is specified, so this is 1>file.txt:

    fd/0 -> pts
    fd/1 -> file.txt
    fd/2 -> pts
    
  2. 2>&1: fd/2 now goes to wherever fd/1 currently goes:

    fd/0 -> pts
    fd/1 -> file.txt
    fd/2 -> file.txt
    

On the other hand, with 2>&1 > file.txt, the order being reversed:

  1. 2>&1: fd/2 now goes to wherever fd/1 currently goes, which means nothing changes:

    fd/0 -> pts
    fd/1 -> pts
    fd/2 -> pts
    
  2. > file.txt: fd/1 now goes to file.txt:

    fd/0 -> pts
    fd/1 -> file.txt
    fd/2 -> pts
    

The important point is that redirection does not mean that the redirected file descriptor will follow all future changes to the target file descriptor; it will only take on the current state.

  • Thank you, this seems like a more natural explanation! :) You made a slight typo though in 2. of the second part; fd/1 -> file.txt and not fd/2 -> file.txt. – Train Heartnet Dec 13 '16 at 14:45
11

I think it helps to think that the shell will set up the redirection on the left first, and complete it before setting up the next redirection.

The Linux Command Line by William Shotts says

First we redirect standard output to the file, and then we redirect file descriptor 2 (standard error) to file descriptor one (standard output)

this makes sense, but then

Notice that the order of the redirections is significant. The redirection of standard error must always occur after redirecting standard output or it doesn't work

but actually, we can redirect stdout to stderr after redirecting stderr to a file with the same effect

$ uname -r 2>/dev/null 1>&2
$ 

So, in command > file 2>&1, the shell sends stdout to a file, then sends stderr to stdout (which is being sent to a file). Whereas, in command 2>&1 > file the shell first redirects stderr to stdout (ie displays it in the terminal where stdout normally goes) and then afterwards, redirects stdout to the file. TLCL is misleading in saying that we must redirect stdout first: since we can redirect stderr to a file first and then send stdout to it. What we cannot do, is redirect stdout to stderr or vice versa before redirection to a file. Another example

$ strace uname -r 1>&2 2> /dev/null 
4.8.0-30-generic

We might think this would dispose of stdout to the same place as stderr, but it doesn't, it redirects stdout to stderr (the screen) first, and then only redirects stderr, as when we tried it the other way round...

I hope this brings a little bit of light...

  • So much more eloquent! – Arronical Dec 13 '16 at 9:44
  • Ah, now I understand! Thank you so much, @Zanna and @Arronical! Just starting out on my command line journey. :) – Train Heartnet Dec 13 '16 at 10:00
  • @TrainHeartnet it's a pleasure! Hope you are enjoying it as much as me :D – Zanna Dec 13 '16 at 10:05
  • @Zanna: Indeed, I am! :D – Train Heartnet Dec 13 '16 at 10:14
  • 2
    @TrainHeartnet no worries, a world of frustration and joy awaits you! – Arronical Dec 13 '16 at 10:27
10
+100

You got a few very good answers already. Let me stress though that there are two different concepts involved here, the understanding of which helps tremendously:

Background: File descriptor vs. file table

Your file descriptor is just a number 0...n, which is the index in the file descriptor table in your process. By convention, STDIN=0, STDOUT=1, STDERR=2 (note that the terms STDIN etc. here are just symbols/macros used by convention in some programming languages and man pages, there is not an actual "object" called STDIN; for the purpose of this discussion, STDIN is 0, etc.).

That file descriptor table in itself does not contain any information whatsoever about what the actual file is. Instead, it contains a pointer to a different file table; the latter contains information about an actual physical file (or block device, or pipe, or whatever else Linux can address via the file mechanism) and more information (i.e., whether it's for reading or writing).

So when you use > or < in your shell, you simply replace the pointer of the respective file descriptor to point to something else. The syntax 2>&1 simply points descriptor 2 to wherever 1 points. > file.txt simply opens file.txt for writing and lets STDOUT (file decsriptor 1) point to that.

There are other goodies, e.g. 2>(xxx) (i.e.: create a new process running xxx, create a pipe, connect file descriptor 0 of the new process to the reading end of the pipe and connect the file descriptor 2 of the original process to the writing end of the pipe).

This is also the basis for "file handle magic" in other software than your shell. For example, you could, in your Perl script, duplicate the STDOUT file descriptor into another (temporary) one, then re-open STDOUT to a newly created temporary file. From this point on, all STDOUT output from your own Perl script and all system() calls of that script will end up in that temporary file. When done, you can re-dup your STDOUT to the temporary descriptor you saved it to, and presto, all is as before. You can even write to that temporary descriptor meanwhile, so while your actual STDOUT output goes to the temporary file, you can still actually output stuff to the real STDOUT (commonly, the user).

Answer

To apply the background information given above to your question:

In what order does the shell execute commands and stream redirection?

Left to right.

<command> > file.txt 2>&1

  1. fork off a new process.
  2. Open file.txt and store its pointer in the file descriptor 1 (STDOUT).
  3. Point STDERR (file descriptor 2) to whatever the fd 1 points to right now (which again is the already opened file.txt of course).
  4. exec the <command>

This apparently redirects stderr to stdout first, and then the resulting stdout is redirected to file.txt.

This would make sense if there were only one table, but as explained above there are two. File descriptors are not pointing to each other recursively, it makes no sense to think "redirect STDERR to STDOUT". The correct thought is "point STDERR to wherever STDOUT points". If you change STDOUT later, STDERR stays where it is, it does not magically go along with further changes to STDOUT.

  • Upvoting for the "file handle magic" bit - although it doesn't directly answer the question I learnt something new today... – Floris Dec 14 '16 at 13:11
3

The order is left to right. Bash's manual has already covered what you ask. Quote from REDIRECTION section of the manual:

   Redirections  are  processed  in  the
   order they appear, from left to right.

and a few lines later:

   Note that the order of redirections is signifi‐
   cant.  For example, the command

          ls > dirlist 2>&1

   directs both standard output and standard error
   to the file dirlist, while the command

          ls 2>&1 > dirlist

   directs   only  the  standard  output  to  file
   dirlist, because the standard error was  dupli‐
   cated from the standard output before the stan‐
   dard output was redirected to dirlist.

It is important to note that redirection is resolved first before any commands run ! See https://askubuntu.com/a/728386/295286

3

It's always left to right... except when

Just like in Math we do left to right except multiplication and division is done before addition and subtraction, except operations inside parenthesis (+ -) would be done before multiplication and division.

As per the Bash beginners guide here (Bash Beginners Guide) there are 8 orders of hierarchy of what comes first (before left to right):

  1. Brace expansion "{}"
  2. Tilde expansion "~"
  3. Shell parameter and variable expression "$"
  4. Command substitution "$(command)"
  5. Arithmetic expression "$((EXPRESSION))"
  6. Process Substitution what we are talking about here "<(LIST)" or ">(LIST)"
  7. Word Splitting "'< space>< tab>< newline>'"
  8. File Name Expansion "*", "?", etc.

So it's always left to right... except when...

  • 1
    To be clear: process substitution and redirection are two independent operations. You can do one without the other. Having process substitution does not mean that bash decides to change the order in which it processes redirection – muru Dec 20 '16 at 0:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.