9

When I use find to see all the pdf files in the /home directory, I am seeing access denied. To eliminate them I tried:

find /home -iname "*.pdf" | grep -v "access denied"

However, the result is the same. How can I get rid of these lines?

1

3 Answers 3

19

What you tried didn't work because the access denied output are errors and sent on STDERR instead of STDOUT which is piped to grep.

You can avoid seeing those errors by redirecting only STDERR

find /home -iname "*.pdf" 2>/dev/null

Or as David Foerster commented we can more succinctly close STDERR

find /home -iname "*.pdf" 2>&-

However, I suspect you actually only want to search your home rather than other users', so perhaps you really want

find ~ -iname "*.pdf"

If that throws errors there may be some wrong ownerships in your local config, which you should investigate.

13
  • 3
    Grrr, why do people always beat me by 30 seconds? :\ Aug 9, 2017 at 11:31
  • find: “/home/ihsan/.gvfs”: access denied find: “/home/ihsan/.dbus”: access denied , for the command ~
    – solfish
    Aug 9, 2017 at 11:34
  • is there something wrong for that? yes i also want to other users home directory which is also created by me to test
    – solfish
    Aug 9, 2017 at 11:34
  • 2
    @solfish As far as I know those files should be owned by you. You may want to sudo chown $USER: ~/.gvfs ~/.dbus
    – Zanna
    Aug 9, 2017 at 11:37
  • 1
    It should be enough to close stderr with 2>&-. GNU find won't terminate itself if it tries to write error messages to a dysfunctional file descriptor. For the ownership issues sudo chown -R $USER: ... would be more effective in case of more files within that aren't owned by $USER. Aug 9, 2017 at 13:56
8

The access denied is probably being printed on stderr rather than stdout.

Try this:

find /home -iname "*.pdf" 2>&1 | grep -v "access denied"

The 2>&1 redirects the output from stderr to stdout, so that grep -v can do its job. (By default, | only pipes stdout and not stderr.)

5
  • but for this 2>&1 means if stderr exists send to stdout?
    – solfish
    Aug 9, 2017 at 11:41
  • @solfish Yup, that's exactly the point :) Aug 9, 2017 at 11:43
  • what i dont understand is before "|" as an output ; we got just stderr right? and after "|" as an input we got this
    – solfish
    Aug 9, 2017 at 11:45
  • @solfish Well, I ran into this problem about a year and a half ago, and I was able to fix it using a different method. But then a comment below my answer suggested to simply use 2>&1... I'm not a bash expert, so if that's incorrect then please do say so :) Aug 9, 2017 at 11:52
  • @AndroidDev I suggest adding that different method to this answer as an alternative. Etan Reisner's criticism was that process substitution isn't portable. But bash in Ubuntu has it, except in POSIX mode. I think it's the best solution--a file maliciously named access denied will still appear. Aug 10, 2017 at 4:20
4

You probably mean "Permission denied"—which is what find in Ubuntu shows you when you cannot access something because of file permissions—rather than "access denied".

One fully general command that does this correctly (and, as a bonus, is portable to other *nixes, so long as the error message is the same) is:

(find 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-

(Usually you want to pass some arguments to find. Those go before the first redirection 3>&1.)

However, often you will able to use something simpler. For example, you can probably use process substitution. Details follow.

The Most Common Methods and Their Limitations

The two typical approaches are to throw away stderr (as in Zanna's answer) or to redirect stderr to stdout and filter stdout (as in Android Dev's answer). Although they have the advantage of being simple to write and are often reasonable choices, these approaches are not ideal.

Discarding everything sent to stderr—such as by redirecting it to the null device with 2>/dev/null or by closing it with 2>&-—runs the risk of missing errors other than "Permission denied".

"Permission denied" is probably the most common error seen when running find, but it is far from the only possible error, and if another does occur, you may want to know about it. In particular, find reports "No such file or directory" if a starting point does not exist. With multiple starting points, find may still return some useful results and appear to work. For example, if a and c exist but b doesn't, find a b c -name x prints results in a, then "No such file or directory" for b, then results in c.

Combining stdout and stderr together into stdout and piping it to grep or some other command to filter it—as with 2>&1 | grep ... or |& grep ...—runs the risk of unintentionally filtering out a file whose name contains the message being filtered.

For example, if you filter out lines that contain "Permission denied" then you will also drop search results showing filenames like "Permission denied messages.txt". This would probably happen by accident, though it would also be possible for a file to be given a specially crafted name to thwart your searches.

Filtering the combined streams has another problem, which cannot be mitigated by filtering more selectively (such as with grep -vx 'find: .*: Permission denied' on the right side of the pipe). Some find actions, including the -print action that is implicit when you specify no action, determine how to output filenames based on whether or not stdout is a terminal.

  • If it isn't a terminal, then the filenames are output as-is even if they contain weird characters like newlines and control characters that could change the behavior of your terminal. If it is a terminal, then these characters are suppressed and ? is printed instead.
  • This is usually what you want. If you are going to process filenames further, they must be output literally. However, if you are going to display them, a filename with a newline could otherwise mimic multiple filenames, and a filename with a sequence of backspace characters could appear to be a different name. Other problems are also possible, such as filenames containing escape sequences that change the colors in your terminal.
  • But piping the search results through another command (like grep) causes find no longer to see a terminal. (More precisely, it causes its stdout not to be a terminal.) Then weird characters are output literally. But if all the command on the right side of the pipe does is (a) remove lines that look like "Permission denied" messages and (b) print what's left, then you're still subject to the sort of shenanigans that find's terminal detection is intended to prevent.
  • See the UNUSUAL FILENAMES section of man find for more information, including the behavior of each of the actions that print filenames. ("Many of the actions of find result in the printing of data which is under the control of other users...") See also sections 3.3.2.1, 3.3.2.2, and 3.3.2.3 of the GNU Findutils reference manual.

The above discussion of unusual filenames pertains to GNU find, which is the find implementation in GNU/Linux systems including Ubuntu.

Leaving Standard Output Alone While Filtering Standard Error

What you really want here is to leave stdout intact while piping stderr to grep. Unfortunately there is no simple syntax for this. | pipes stdout, and some shells (including bash) support |& to pipe both streams—or you can redirect stderr to stdout first with 2>&1 |, which has the same effect. But the commonly used shells don't provide a syntax to pipe stderr only.

You can still do this. It's just awkward. One way is to swap stdout with stderr, so that search results are on stderr and errors are on stdout, then pipe stdout to grep for filtering:

find args 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied'

Usually you'll pass arguments to find, such as starting points (the places to search from, which are usually directories) and predicates (tests and actions). These go in place of args above.

This works by introducing a new file descriptor to hold onto one of the two standard streams you want to swap, performing redirections to swap them, and closing the new file descriptor.

  • File descriptor 1 is stdout and 2 is stderr (and the unredirected 0 is stdin). But you can also redirect using other file descriptors. This can be used to open, or keep open, a file or device.
  • 3>&1 redirects file descriptor 3 to stdout, so that when stdout (file descriptor 1) is subsequently redirected, the original stdout can still be written to easily.
  • 1>&2 redirects stdout to stderr. Since file descriptor 3 is still the original stdout, it can still be accessed.
  • 2>&3 redirects stderr to file descriptor 3, which is the original stdout.
  • 3>&- closes file descriptor 3, which is no longer needed.
  • For more information, see How to pipe stderr, and not stdout? and IO Redirection - Swapping stdout and stderr (Advanced) and especially pipe only stderr through a filter.

However, this method has the disadvantage that search results are sent to stderr and errors are sent to stdout. If you're running this command directly in an interactive shell and not piping or redirecting the output any further, then that doesn't really matter. Otherwise, it may be a problem. If you put that command in a script, and then someone (perhaps you, later) redirects or pipes its output, it does not behave as expected.

The solution is to swap the streams back after you're done filtering the output. Applying the same redirections shown above on the right side of the pipeline won't achieve this, because | only pipes stdout, so that side of the pipeline only receives output that was originally sent to stderr (because the streams were swapped) and not the original stdout output. Instead, you can use ( ) to run the above command in a subshell (related), then apply the swapping redirections to that:

(find args 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-

It's the grouping, not specifically the subshell, that makes this work. If you prefer, you can use { ;}:

{ find args 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied'; } 3>&1 1>&2 2>&3 3>&-

A Less Cumbersome Way: Process Substitution

Some shells, including Bash on systems that can support it (including GNU/Linux systems like Ubuntu), let you perform process substitution, which allows you to run a command and redirect to/from one of its streams. You can redirect the find command's stderr to a grep command that filters it, and redirect that grep command's stdout to stderr.

find args 2> >(grep -Fv 'Permission denied' >&2)

Credit goes to to Android Dev for this idea.

Although bash supports process substitution, sh in Ubuntu is dash, which doesn't. It will give you "Syntax error: redirection unexpected" if you try to use this method, whereas the method of swapping stdout and stderr will still work. Furthermore, when bash runs in POSIX mode, support for process substitution is turned off.

One situation where bash runs in POSIX mode is when it is invoked as sh1. Therefore, on an OS like Fedora where bash provides /bin/sh, or if you have made the /bin/sh symlink point to bash yourself on Ubuntu, process substitution still does not work in an sh script, without a prior command to turn POSIX mode off. Your best bet, if you want to use this method in a script, is to put #!/bin/bash at the top instead of #!/bin/sh, if you aren't already.

1: In this situation, bash turns on POSIX mode automatically after it runs the commands in its startup scripts.

An Example

It is useful to be able to test these commands. To do this, I create a tmp subdirectory of the current directory and populate it with some files and directories, taking permissions away from one of them to trigger a "Permission denied" error in find.

mkdir tmp; cd tmp; mkdir a b c; touch w a/x 'a/Permission denied messages.txt' b/y c/z; chmod 0 b

One of the directories that is accessible includes a file with "Permission denied" in its name. Running find with no redirections or pipes shows this file, but also shows the actual "Permission denied" error for another directory that isn't accessible:

ek@Io:~/tmp$ find
.
./a
./a/Permission denied messages.txt
./a/x
./c
./c/z
./w
./b
find: ‘./b’: Permission denied

Piping both stdout and stderr to grep and filtering out lines that contain "Permission denied" makes the error message go away but also hides the search result for the file with that phrase in its name:

ek@Io:~/tmp$ find |& grep -Fv 'Permission denied'
.
./a
./a/x
./c
./c/z
./w
./b

find 2>&1 | grep -Fv 'Permission denied' is equivalent and produces the same output.

The methods shown above for filtering out "Permission denied" only from error messages—and not from search results—are successful. For example, here's the method where stdout and stderr are swapped:

ek@Io:~/tmp$ (find 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-
.
./a
./a/Permission denied messages.txt
./a/x
./c
./c/z
./w
./b

find args 2> >(grep -Fv 'Permission denied' >&2) produces the same output.

You can trigger a different error message to ensure that lines sent to stderr that don't contain the text "Permission denied" are still allowed through. For example, here I have run find with the current directory (.) as one starting point, but the nonexistent directory foo as another:

ek@Io:~/tmp$ (find . foo 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-
.
./a
./a/Permission denied messages.txt
./a/x
./c
./c/z
./w
./b
find: ‘foo’: No such file or directory

Checking That find's Standard Output is Still a Terminal

We can also see which commands cause special characters, such as newlines, to be displayed literally. (This can be done separately from the demonstration above, and it doesn't need to be in the tmp directory.)

Make a file with a newline in its name:

touch $'abc\ndef'

Usually we use directories as starting points for find, but files work too:

$ find abc*
abc?def

Piping stdout to another command causes the newline to be outputted literally, creating the false impression of two separate search results abc and def. We can test that with cat:

$ find abc* | cat
abc
def

Redirecting just stderr does not cause this problem:

$ find abc* 2>/dev/null
abc?def

Nor does closing it:

$ find abc* 2>&-
abc?def

Piping to grep does cause the problem:

$ find abc* |& grep -Fv 'Permission denied'
abc
def

(Replacing |& with 2>&1 | is equivalent and produces the same output.)

Swapping stdout and stderr and piping stdout does not cause the problem—find's stdout becomes stderr, which is not piped:

$ find abc* 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied'
abc?def

Grouping that command and swapping the streams back does not cause the problem:

$ (find abc* 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-
abc?def

(The { ;} version produces the same output.)

Using process substitution to filter stderr does not cause the problem either:

$ find abc* 2> >(grep -Fv 'Permission denied' >&2)
abc?def

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .