In what order does the shell execute commands and stream redirection?

Question

I was trying to redirect both stdout and stderr to a file today, and I came across this:

<command> > file.txt 2>&1

This apparently redirects stderr to stdout first, and then the resulting stdout is redirected to file.txt.

However, why isn't the order <command> 2>&1 > file.txt? One would naturally read this as (assuming execution from left to right) the command being executed first, the stderr being redirected to stdout and then, the resulting stdout being written to file.txt. But the above only redirects stderr to the screen.

How does the shell interpret both the commands?

Answer 1

When you run <command> 2>&1 > file.txt stderr is redirected by 2>&1 to where stdout currently goes, your terminal. After that, stdout is redirected to the file by >, but stderr is not redirected with it, so stays as terminal output.

With <command> > file.txt 2>&1 stdout is first redirected to the file by >, then 2>&1 redirects stderr to where stdout is going, which is the file.

It may seem counter intuitive to start with, but when you think of the redirections in this way, and remember that they are processed from left to right it makes much more sense.

Answer 2

It might make sense if you trace it out.

In the beginning, stderr and stdout go to the same thing (usually the terminal, which I call here pts):

fd/0 -> pts
fd/1 -> pts
fd/2 -> pts

I'm referring to stdin, stdout and stderr by their file descriptor numbers here: they are file descriptors 0, 1 and 2 respectively.

Now, in the first set of redirections, we have > file.txt and 2>&1.

So:

1. > file.txt: fd/1 now goes to file.txt. With >, 1 is the implied file descriptor when nothing is specified, so this is 1>file.txt:

   fd/0 -> pts
   fd/1 -> file.txt
   fd/2 -> pts
   

2. 2>&1: fd/2 now goes to wherever fd/1 currently goes:

   fd/0 -> pts
   fd/1 -> file.txt
   fd/2 -> file.txt
   

On the other hand, with 2>&1 > file.txt, the order being reversed:

1. 2>&1: fd/2 now goes to wherever fd/1 currently goes, which means nothing changes:

   fd/0 -> pts
   fd/1 -> pts
   fd/2 -> pts
   

2. > file.txt: fd/1 now goes to file.txt:

   fd/0 -> pts
   fd/1 -> file.txt
   fd/2 -> pts
   

The important point is that redirection does not mean that the redirected file descriptor will follow all future changes to the target file descriptor; it will only take on the current state.

Answer 3

I think it helps to think that the shell will set up the redirection on the left first, and complete it before setting up the next redirection.

The Linux Command Line by William Shotts says

First we redirect standard output to the file, and then we redirect file descriptor 2 (standard error) to file descriptor one (standard output)

this makes sense, but then

Notice that the order of the redirections is significant. The redirection of standard error must always occur after redirecting standard output or it doesn't work

but actually, we can redirect stdout to stderr after redirecting stderr to a file with the same effect

$ uname -r 2>/dev/null 1>&2
$ 

So, in command > file 2>&1, the shell sends stdout to a file, then sends stderr to stdout (which is being sent to a file). Whereas, in command 2>&1 > file the shell first redirects stderr to stdout (ie displays it in the terminal where stdout normally goes) and then afterwards, redirects stdout to the file. TLCL is misleading in saying that we must redirect stdout first: since we can redirect stderr to a file first and then send stdout to it. What we cannot do, is redirect stdout to stderr or vice versa before redirection to a file. Another example

$ strace uname -r 1>&2 2> /dev/null 
4.8.0-30-generic

We might think this would dispose of stdout to the same place as stderr, but it doesn't, it redirects stdout to stderr (the screen) first, and then only redirects stderr, as when we tried it the other way round...

I hope this brings a little bit of light...

Answer 4

You got a few very good answers already. Let me stress though that there are two different concepts involved here, the understanding of which helps tremendously:

Background: File descriptor vs. file table

Your file descriptor is just a number 0...n, which is the index in the file descriptor table in your process. By convention, STDIN=0, STDOUT=1, STDERR=2 (note that the terms STDIN etc. here are just symbols/macros used by convention in some programming languages and man pages, there is not an actual "object" called STDIN; for the purpose of this discussion, STDIN is 0, etc.).

That file descriptor table in itself does not contain any information whatsoever about what the actual file is. Instead, it contains a pointer to a different file table; the latter contains information about an actual physical file (or block device, or pipe, or whatever else Linux can address via the file mechanism) and more information (i.e., whether it's for reading or writing).

So when you use > or < in your shell, you simply replace the pointer of the respective file descriptor to point to something else. The syntax 2>&1 simply points descriptor 2 to wherever 1 points. > file.txt simply opens file.txt for writing and lets STDOUT (file decsriptor 1) point to that.

There are other goodies, e.g. 2>(xxx) (i.e.: create a new process running xxx, create a pipe, connect file descriptor 0 of the new process to the reading end of the pipe and connect the file descriptor 2 of the original process to the writing end of the pipe).

This is also the basis for "file handle magic" in other software than your shell. For example, you could, in your Perl script, duplicate the STDOUT file descriptor into another (temporary) one, then re-open STDOUT to a newly created temporary file. From this point on, all STDOUT output from your own Perl script and all system() calls of that script will end up in that temporary file. When done, you can re-dup your STDOUT to the temporary descriptor you saved it to, and presto, all is as before. You can even write to that temporary descriptor meanwhile, so while your actual STDOUT output goes to the temporary file, you can still actually output stuff to the real STDOUT (commonly, the user).

Answer

To apply the background information given above to your question:

In what order does the shell execute commands and stream redirection?

Left to right.

<command> > file.txt 2>&1

1. fork off a new process.
2. Open file.txt and store its pointer in the file descriptor 1 (STDOUT).
3. Point STDERR (file descriptor 2) to whatever the fd 1 points to right now (which again is the already opened file.txt of course).
4. exec the <command>

This apparently redirects stderr to stdout first, and then the resulting stdout is redirected to file.txt.

This would make sense if there were only one table, but as explained above there are two. File descriptors are not pointing to each other recursively, it makes no sense to think "redirect STDERR to STDOUT". The correct thought is "point STDERR to wherever STDOUT points". If you change STDOUT later, STDERR stays where it is, it does not magically go along with further changes to STDOUT.

Answer 5

The order is left to right. Bash's manual has already covered what you ask. Quote from REDIRECTION section of the manual:

   Redirections  are  processed  in  the
   order they appear, from left to right.

and a few lines later:

   Note that the order of redirections is signifi‐
   cant.  For example, the command

          ls > dirlist 2>&1

   directs both standard output and standard error
   to the file dirlist, while the command

          ls 2>&1 > dirlist

   directs   only  the  standard  output  to  file
   dirlist, because the standard error was  dupli‐
   cated from the standard output before the stan‐
   dard output was redirected to dirlist.

It is important to note that redirection is resolved first before any commands run ! See https://askubuntu.com/a/728386/295286

Answer 6

It's always left to right... except when

Just like in Math we do left to right except multiplication and division is done before addition and subtraction, except operations inside parenthesis (+ -) would be done before multiplication and division.

As per the Bash beginners guide here (Bash Beginners Guide) there are 8 orders of hierarchy of what comes first (before left to right):

1. Brace expansion "{}"
2. Tilde expansion "~"
3. Shell parameter and variable expression "$"
4. Command substitution "$(command)"
5. Arithmetic expression "$((EXPRESSION))"
6. Process Substitution what we are talking about here "<(LIST)" or ">(LIST)"
7. Word Splitting "'< space>< tab>< newline>'"
8. File Name Expansion "*", "?", etc.

So it's always left to right... except when...