When I use find
to see all the pdf files in the /home
directory, I am seeing access denied
. To eliminate them I tried:
find /home -iname "*.pdf" | grep -v "access denied"
However, the result is the same. How can I get rid of these lines?
When I use find
to see all the pdf files in the /home
directory, I am seeing access denied
. To eliminate them I tried:
find /home -iname "*.pdf" | grep -v "access denied"
However, the result is the same. How can I get rid of these lines?
What you tried didn't work because the access denied
output are errors and sent on STDERR instead of STDOUT which is piped to grep
.
You can avoid seeing those errors by redirecting only STDERR
find /home -iname "*.pdf" 2>/dev/null
Or as David Foerster commented we can more succinctly close STDERR
find /home -iname "*.pdf" 2>&-
However, I suspect you actually only want to search your home rather than other users', so perhaps you really want
find ~ -iname "*.pdf"
If that throws errors there may be some wrong ownerships in your local config, which you should investigate.
sudo chown $USER: ~/.gvfs ~/.dbus
2>&-
. GNU find won't terminate itself if it tries to write error messages to a dysfunctional file descriptor. For the ownership issues sudo chown -R $USER: ...
would be more effective in case of more files within that aren't owned by $USER
.
Aug 9, 2017 at 13:56
The access denied is probably being printed on stderr
rather than stdout
.
Try this:
find /home -iname "*.pdf" 2>&1 | grep -v "access denied"
The 2>&1
redirects the output from stderr
to stdout
, so that grep -v
can do its job. (By default, |
only pipes stdout
and not stderr
.)
2>&1
... I'm not a bash expert, so if that's incorrect then please do say so :)
Aug 9, 2017 at 11:52
bash
in Ubuntu has it, except in POSIX mode. I think it's the best solution--a file maliciously named access denied
will still appear.
Aug 10, 2017 at 4:20
You probably mean "Permission denied"—which is what find
in Ubuntu shows you when you cannot access something because of file permissions—rather than "access denied".
One fully general command that does this correctly (and, as a bonus, is portable to other *nixes, so long as the error message is the same) is:
(find 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-
(Usually you want to pass some arguments to find
. Those go before the first redirection 3>&1
.)
However, often you will able to use something simpler. For example, you can probably use process substitution. Details follow.
The two typical approaches are to throw away stderr (as in Zanna's answer) or to redirect stderr to stdout and filter stdout (as in Android Dev's answer). Although they have the advantage of being simple to write and are often reasonable choices, these approaches are not ideal.
Discarding everything sent to stderr—such as by redirecting it to the null device with 2>/dev/null
or by closing it with 2>&-
—runs the risk of missing errors other than "Permission denied".
"Permission denied" is probably the most common error seen when running find
, but it is far from the only possible error, and if another does occur, you may want to know about it. In particular, find
reports "No such file or directory" if a starting point does not exist. With multiple starting points, find
may still return some useful results and appear to work. For example, if a
and c
exist but b
doesn't, find a b c -name x
prints results in a
, then "No such file or directory" for b
, then results in c
.
Combining stdout and stderr together into stdout and piping it to grep
or some other command to filter it—as with 2>&1 | grep ...
or |& grep ...
—runs the risk of unintentionally filtering out a file whose name contains the message being filtered.
For example, if you filter out lines that contain "Permission denied" then you will also drop search results showing filenames like "Permission denied messages.txt". This would probably happen by accident, though it would also be possible for a file to be given a specially crafted name to thwart your searches.
Filtering the combined streams has another problem, which cannot be mitigated by filtering more selectively (such as with grep -vx 'find: .*: Permission denied'
on the right side of the pipe). Some find
actions, including the -print
action that is implicit when you specify no action, determine how to output filenames based on whether or not stdout is a terminal.
?
is printed instead.grep
) causes find
no longer to see a terminal. (More precisely, it causes its stdout not to be a terminal.) Then weird characters are output literally. But if all the command on the right side of the pipe does is (a) remove lines that look like "Permission denied" messages and (b) print what's left, then you're still subject to the sort of shenanigans that find
's terminal detection is intended to prevent.man find
for more information, including the behavior of each of the actions that print filenames. ("Many of the actions of find result in the printing of data which is under the control of other users...") See also sections 3.3.2.1, 3.3.2.2, and 3.3.2.3 of the GNU Findutils reference manual.The above discussion of unusual filenames pertains to GNU find, which is the find
implementation in GNU/Linux systems including Ubuntu.
What you really want here is to leave stdout intact while piping stderr to grep
. Unfortunately there is no simple syntax for this. |
pipes stdout, and some shells (including bash
) support |&
to pipe both streams—or you can redirect stderr to stdout first with 2>&1 |
, which has the same effect. But the commonly used shells don't provide a syntax to pipe stderr only.
You can still do this. It's just awkward. One way is to swap stdout with stderr, so that search results are on stderr and errors are on stdout, then pipe stdout to grep
for filtering:
find args 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied'
Usually you'll pass arguments to find
, such as starting points (the places to search from, which are usually directories) and predicates (tests and actions). These go in place of args
above.
This works by introducing a new file descriptor to hold onto one of the two standard streams you want to swap, performing redirections to swap them, and closing the new file descriptor.
3>&1
redirects file descriptor 3 to stdout, so that when stdout (file descriptor 1) is subsequently redirected, the original stdout can still be written to easily.1>&2
redirects stdout to stderr. Since file descriptor 3 is still the original stdout, it can still be accessed.2>&3
redirects stderr to file descriptor 3, which is the original stdout.3>&-
closes file descriptor 3, which is no longer needed.However, this method has the disadvantage that search results are sent to stderr and errors are sent to stdout. If you're running this command directly in an interactive shell and not piping or redirecting the output any further, then that doesn't really matter. Otherwise, it may be a problem. If you put that command in a script, and then someone (perhaps you, later) redirects or pipes its output, it does not behave as expected.
The solution is to swap the streams back after you're done filtering the output. Applying the same redirections shown above on the right side of the pipeline won't achieve this, because |
only pipes stdout, so that side of the pipeline only receives output that was originally sent to stderr (because the streams were swapped) and not the original stdout output. Instead, you can use (
)
to run the above command in a subshell (related), then apply the swapping redirections to that:
(find args 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-
It's the grouping, not specifically the subshell, that makes this work. If you prefer, you can use {
;}
:
{ find args 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied'; } 3>&1 1>&2 2>&3 3>&-
Some shells, including Bash on systems that can support it (including GNU/Linux systems like Ubuntu), let you perform process substitution, which allows you to run a command and redirect to/from one of its streams. You can redirect the find
command's stderr to a grep
command that filters it, and redirect that grep
command's stdout to stderr.
find args 2> >(grep -Fv 'Permission denied' >&2)
Credit goes to to Android Dev for this idea.
1>&2
. I have used >&2
, which is equivalent (related). Use whichever you like.Although bash
supports process substitution, sh
in Ubuntu is dash
, which doesn't. It will give you "Syntax error: redirection unexpected" if you try to use this method, whereas the method of swapping stdout and stderr will still work. Furthermore, when bash
runs in POSIX mode, support for process substitution is turned off.
One situation where bash
runs in POSIX mode is when it is invoked as sh
1. Therefore, on an OS like Fedora where bash
provides /bin/sh
, or if you have made the /bin/sh
symlink point to bash
yourself on Ubuntu, process substitution still does not work in an sh
script, without a prior command to turn POSIX mode off. Your best bet, if you want to use this method in a script, is to put #!/bin/bash
at the top instead of #!/bin/sh
, if you aren't already.
1: In this situation, bash
turns on POSIX mode automatically after it runs the commands in its startup scripts.
It is useful to be able to test these commands. To do this, I create a tmp
subdirectory of the current directory and populate it with some files and directories, taking permissions away from one of them to trigger a "Permission denied" error in find
.
mkdir tmp; cd tmp; mkdir a b c; touch w a/x 'a/Permission denied messages.txt' b/y c/z; chmod 0 b
One of the directories that is accessible includes a file with "Permission denied" in its name. Running find
with no redirections or pipes shows this file, but also shows the actual "Permission denied" error for another directory that isn't accessible:
ek@Io:~/tmp$ find
.
./a
./a/Permission denied messages.txt
./a/x
./c
./c/z
./w
./b
find: ‘./b’: Permission denied
Piping both stdout and stderr to grep
and filtering out lines that contain "Permission denied" makes the error message go away but also hides the search result for the file with that phrase in its name:
ek@Io:~/tmp$ find |& grep -Fv 'Permission denied'
.
./a
./a/x
./c
./c/z
./w
./b
find 2>&1 | grep -Fv 'Permission denied'
is equivalent and produces the same output.
The methods shown above for filtering out "Permission denied" only from error messages—and not from search results—are successful. For example, here's the method where stdout and stderr are swapped:
ek@Io:~/tmp$ (find 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-
.
./a
./a/Permission denied messages.txt
./a/x
./c
./c/z
./w
./b
find args 2> >(grep -Fv 'Permission denied' >&2)
produces the same output.
You can trigger a different error message to ensure that lines sent to stderr that don't contain the text "Permission denied" are still allowed through. For example, here I have run find
with the current directory (.
) as one starting point, but the nonexistent directory foo
as another:
ek@Io:~/tmp$ (find . foo 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-
.
./a
./a/Permission denied messages.txt
./a/x
./c
./c/z
./w
./b
find: ‘foo’: No such file or directory
find
's Standard Output is Still a TerminalWe can also see which commands cause special characters, such as newlines, to be displayed literally. (This can be done separately from the demonstration above, and it doesn't need to be in the tmp
directory.)
Make a file with a newline in its name:
touch $'abc\ndef'
Usually we use directories as starting points for find
, but files work too:
$ find abc*
abc?def
Piping stdout to another command causes the newline to be outputted literally, creating the false impression of two separate search results abc
and def
. We can test that with cat
:
$ find abc* | cat
abc
def
Redirecting just stderr does not cause this problem:
$ find abc* 2>/dev/null
abc?def
Nor does closing it:
$ find abc* 2>&-
abc?def
Piping to grep
does cause the problem:
$ find abc* |& grep -Fv 'Permission denied'
abc
def
(Replacing |&
with 2>&1 |
is equivalent and produces the same output.)
Swapping stdout and stderr and piping stdout does not cause the problem—find
's stdout becomes stderr, which is not piped:
$ find abc* 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied'
abc?def
Grouping that command and swapping the streams back does not cause the problem:
$ (find abc* 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-
abc?def
(The {
;}
version produces the same output.)
Using process substitution to filter stderr does not cause the problem either:
$ find abc* 2> >(grep -Fv 'Permission denied' >&2)
abc?def