grep recursively for specific files

Question

I tried to use grep to find some code snippets in python files which are spreaded over some directories / subdirectories, but unfortunately my attempts failed :(

I thought:

grep -r "search-pattern" *.py

should do the magic, but it failed with "no matches found", although there are several files containing lines with the search pattern.

Next I tried the following:

grep -r "search-pattern" .

Which seemed to worked, but also returned many errors for some compiled c-files and stuff. Obviously more than I wanted.

Finally, after many Google searches, I came up with:

grep -rn --include="*.py" "search-pattern"

This did the job and found all python files I was searching for. As a bonus it also prints the line numbers containing the search-pattern.

But one problem remained: some "permission denied" errors. How do I get rid of those? I thought handling grep would be easy but it turned out being complex to get nice results, with line numbering and without errors..

Any help would be highly appreciated :)

Answer 1

You are getting these Permission denied errors because there are some python files that your user does not have permission to read, so grep is not working for those files (you can't search them).

You want to "get rid of" the Permission denied errors.

1. You can add 2>/dev/null to the end of your command like this:

   grep -rn --include="*.py" "search-pattern" 2>/dev/null
   

   This suppresses all error messages, so you won't see the Permission denied errors popping up. But this obviously still means that grep is not searching those files, so if they do contain your search-pattern, you won't see them in the output.

   Explanation:
   In Linux, there are three things called file descriptors: stdin, stdout, stderr. If you run a command and it gives you an error, the error is written to the file descriptor stderr. By default, stderr outputs the errors to the terminal. So what we can do is redirect the errors, i.e. we redirect stderr somewhere else instead of the terminal. We redirect stuff in Linux by using the > sign. So where are we going to redirect stderr? To /dev/null. /dev/null is a special file; think of it as a black hole. Anything you redirect to it gets thrown away. Finally, we refer to stderr as simply the number 2. So 2>/dev/null tells Linux to redirect 2 to /dev/null. 1 means stdout (which is the normal output of a command, i.e. not errors). So, for example, if you wanted to save the actual output of your grep command to a file rather than have it displayed in the terminal, you can use 1>/path/to/filename, which will redirect stdout to that file.

2. Use sudo with your command, assuming that you have the privileges:

   sudo grep -rn --include="*.py" "search-pattern"
   

   If your username has root privileges (i.e. if you're an administrator on the machine), using sudo before a command runs it as the root user, so you will never get the Permission denied errors and grep will search those files and output the search-pattern if it finds it inside them.

Answer 2

It's a common problem, you can solve it directly from command line.
You can use --devices=skip to ignore the special device files (source of errors and endless loops on many systems).
Execute sudo chown -R new_owner_name file_pattern to change the file ownership recursively and avoid the permission errors.
If you redirect the error stream with COMMAND 2> DESTINATION_OR_/dev/null the errors still happen and you won't get the desired results from grep.
If possible never use sudo automatically to do the analysis (it can be a weak spot of the system), try to change the file ownership when needed. A graphical version of sudo is available if you need to ask the user for a permission.

Please, feel free to comment under here if you have more inherent questions and don't forget to press the left UP arrow if I'm of any help.
Have a nice day.