What are these auxiliary file descriptors?

Question

While poking around /proc/self folder for my mksh shell, I found a peculiar thing: in /proc/self/fd/* there are all the standard file descriptors (0 for stdin, 1 for stdout, and 2 stderr) some file descriptors, but also some extra ones - 24, 25, 3. And I technically can list them with a glob in the shell:

$ for fd in /proc/self/fd/* ; do echo $fd ; done                                                      
/proc/self/fd/0
/proc/self/fd/1
/proc/self/fd/2
/proc/self/fd/24
/proc/self/fd/25
/proc/self/fd/3

But when I try to stat them or use find on them, they are reported as non-existent.

$ find  /proc/self/fd/*                                                                               
/proc/self/fd/0
/proc/self/fd/1
/proc/self/fd/2
find: ‘/proc/self/fd/24’: No such file or directory
find: ‘/proc/self/fd/25’: No such file or directory
/proc/self/fd/3

Same occurs in bash , but just with just one auxiliary file descriptor.

$ for fd in /proc/self/fd/* ; do echo $fd; done
/proc/self/fd/0
/proc/self/fd/1
/proc/self/fd/2
/proc/self/fd/255
/proc/self/fd/3

$ find /proc/self/fd/*
/proc/self/fd/0
/proc/self/fd/1
/proc/self/fd/2
find: ‘/proc/self/fd/255’: No such file or directory
/proc/self/fd/3

Question is: what are those additional file descriptors? What is their purpose?

Answer 1

Probing /proc/self is a tricky business, since it changes for each process. When you do /proc/self/fd/*, the shell expands the wildcard, so it's listing its own file descriptors. But when these are passed to another command, like find or ls, the paths will now be for that process' /proc/self, and it may or may not have fds with those numbers.

Even trickier, the shell may open file descriptors during wildcard expansion.

Comparing with /proc/$$/fd might be illuminating:

bash:

$ ls -l /proc/self/fd /proc/$$/fd/* &
[1] 5172
$ lrwx------ 1 muru muru 64 Jan  1 20:16 /proc/4932/fd/0 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  1 20:16 /proc/4932/fd/1 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  1 20:16 /proc/4932/fd/2 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  1 20:16 /proc/4932/fd/255 -> /dev/pts/1

/proc/self/fd:
total 0
lrwx------ 1 muru muru 64 Jan  1 20:24 0 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  1 20:24 1 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  1 20:24 2 -> /dev/pts/1
lr-x------ 1 muru muru 64 Jan  1 20:24 3 -> /proc/5172/fd

[1]+  Done                    ls --color=auto -l /proc/self/fd /proc/$$/fd/*

By sending it to the background, I got bash to print the PID, and you can see that /proc/self/fd/3 points to ls' own /proc/<PID>/fd, which it had opened to scan. The entries with 4932, OTOH, are for bash's fds, and the special one is 255. An explanation is found in this SO post:

The open files are 0 (stdin), 1 (stdout), and 2 (stderr). 255 is a little trick that bash uses to keep a copy of these for when they are redirected. This is specific to bash.

Source: https://books.google.com/books?id=wWjqCF9HLfYC&pg=PA231

With mksh:

$ ls -l /proc/self/fd /proc/$$/fd/*   &
[1] 5075
$ lrwx------ 1 muru muru 64 Jan  1 20:22 /proc/5074/fd/0 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  1 20:22 /proc/5074/fd/1 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  1 20:22 /proc/5074/fd/10 -> /dev/tty
lrwx------ 1 muru muru 64 Jan  1 20:22 /proc/5074/fd/2 -> /dev/pts/1

/proc/self/fd:
total 0
lrwx------ 1 muru muru 64 Jan  1 20:22 0 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  1 20:22 1 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  1 20:22 2 -> /dev/pts/1
lr-x------ 1 muru muru 64 Jan  1 20:22 3 -> /proc/5075/fd

[1] + Done                 ls -l /proc/self/fd /proc/$$/fd/* 

Practically the same thing, except that the extra fd is 10, and I'd wager it's for the same reason as bash, since the source code indicates that fd 10 and onwards is used by the shell.

I didn't get two or three extra fds, but it could be due to any number of things happening during wildcard expansion, or due to background jobs or some other obscure reason.

If I run your for loop, I do get an ephemeral fd 3:

$ for fd in /proc/$$/fd/* ; do ls -l $fd ; done
lrwx------ 1 muru muru 64 Jan  2 17:39 /proc/6012/fd/0 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  2 17:39 /proc/6012/fd/1 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  2 17:39 /proc/6012/fd/2 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  2 17:39 /proc/6012/fd/255 -> /dev/pts/1
ls: cannot access '/proc/6012/fd/3': No such file or directory

And here, using strace to trace the execution:

strace -e open -o log bash -c 'for fd in /proc/$$/fd/* ; do : ; done'

We'll see that the third fd is, in fact, /proc/<PID>/fd:

$ tail log
open("/usr/lib/libreadline.so.7", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/libncursesw.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/dev/tty", O_RDWR|O_NONBLOCK)     = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) = 3
open("/proc/9975/fd/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
+++ exited with 0 +++

Now the question is, why didn't this fd show up in the earlier ls tests? It looks like backgrounding had something to do with this:

$ ls -l /proc/$$/fd/*   &
[1] 10091
$ lrwx------ 1 muru muru 64 Jan  2 17:46 /proc/10076/fd/0 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  2 17:46 /proc/10076/fd/1 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  2 17:46 /proc/10076/fd/2 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  2 17:46 /proc/10076/fd/255 -> /dev/pts/1

[1]+  Done                    ls --color=auto -l /proc/self/fd /proc/$$/fd/*
$ ls -l /proc/$$/fd/* 
ls: cannot access '/proc/10076/fd/3': No such file or directory
lrwx------ 1 muru muru 64 Jan  2 17:46 /proc/10076/fd/0 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  2 17:46 /proc/10076/fd/1 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  2 17:46 /proc/10076/fd/2 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  2 17:46 /proc/10076/fd/255 -> /dev/pts/1

The foreground ls shows the missing fd.

Now, again tracing with strace:

strace -fe open,execve,fork -o log bash -ic 'ls -l /proc/self/fd /proc/$$/fd/* &'

We see:

10731 execve("/usr/bin/bash", ["bash", "-ic", "ls -l /proc/$$/fd/* &"], [/* 67 vars */]) = 0
10731 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
# snip
10734 open("/proc/10731/fd/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
10734 execve("/usr/bin/ls", ["ls", "--color=auto", "-l", "/proc/10731/fd/0", "/proc/10731/fd/1", "/proc/10731/fd/2", "/proc/10731/fd/255"], [/* 68 vars */]) = 0

Note the change in PIDs. It seems that wildcard expansion takes place after forking, but variable expansion happens before that. So, fd 3 exists, but in a different process. Now i you use self instead of $$, you'll see both 3 and 255:

$ strace -fe open,execve -o log bash -ic 'ls -l /proc/self/fd/* &'
[1] 10790
ls: cannot access '/proc/self/fd/255': No such file or directory
ls: cannot access '/proc/self/fd/3': No such file or directory
lrwx------ 1 muru muru 64 Jan  2 18:04 /proc/self/fd/0 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  2 18:04 /proc/self/fd/1 -> /dev/pts/1
lrwx------ 1 muru muru 64 Jan  2 18:04 /proc/self/fd/2 -> /dev/pts/1

Addendum

A related answer on Unix&Linux Stackexchange site cites a response from a mail list:

Fd 255 is used internally as a connection to the tty, so that it doesn't interfere with the use of exec to relocate fds. Bash also allocates high fds when handling a process substitution `<(foo)', for the same reason.

Andreas Schwab