3

New question on 22.9.2016, down below!

My path looks like this:

~/Desktop/logs
├── first_folder
|   ├── 11.11 (folder)
│   |   ├── access_log
│   |   ├── access_log.1.gz
│   |   :
│   |   └── access_log.40.gz
|   └── 11.12 (folder)
│       ├── access_log
│       ├── access_log.1.gz
│       :
│       └── access_log.16.gz
├── second_folder (folder)
|   ├── 31.11 (folder)
│   |   ├── access_log
│   |   ├── access_log.1.gz
│   |   :
│   |   └── access_log.20.gz
|   └── 31.15 (folder)
│       ├── access_log
│       ├── access_log.1.gz
│       :
:       └── access_log.38.gz
└── last_folder
    ├── 91.11 (folder)
    |   ├── access_log
    |   ├── access_log.1.gz
    |   :
    |   └── access_log.25.gz
    └── 91.15 (folder)
        ├── access_log
        ├── access_log.1.gz
        :
        └── access_log.30.gz

From all the logs I have to extract the data that contain string: /Jan/2016

Question #1

How can I get all the records from all the files in all folders and save it as single_file.txt

Edit #1

Possible answer:

$ find . -name \*.* | xargs -0 zgrep -E '/Jan/2016' > single_file.txt

File is really really huge, lots of GB.

Output:

single_file.txt
./first_folder/11.11/access_log.9.gz: ... text ...
./first_folder/11.12/access_log.9.gz: ... text ...
./second_folder/31.11/access_log.9.gz: ... text ...
./second_folder/31.11/access_log.9.gz: ... text ...
:

Question #2

How can I get all the records from all the files in all folders and save them individually depending on records I grab from first_folder, second_folder etc like first.txt, second.txt.

Output would look something like:

first.txt
./first_folder/11.11/access_log.9.gz: ... text ...
./first_folder/11.12/access_log.9.gz: ... text ...
:
second.txt
./second_folder/31.11/access_log.9.gz: ... text ...
./second_folder/31.15/access_log.9.gz: ... text ...
:

Question #3

How can I get list of all the files that contain string /Jan/2016?

If I use this command, will it retrieve all the files:

$ find ~/Desktop/logs/ -type f | xargs zgrep -l "/Jan/2016"  

Output:

Terminal
/home/name/Desktop/logs/first_folder/11.11/access_log.9.gz
/home/name/Desktop/logs/first_folder/11.12/access_log.8.gz
/home/name/Desktop/logs/second_folder/31.11/access_log.6.gz
:

Edit #2

@waltinator-s code and @Zanna-s correction:

pushd ~/Desktop/logs
for dir in * ; do
    if [[ -d "$dir" ]] ; then
        outname="$dir.txt"
        find "$dir" -type f -print0 | xargs -0 zgrep -l '/Jan/2016' >"$outname"
    fi
done
popd

will give me structure:

~/Desktop/logs
├── first_folder
|   └── first.txt
├── second_folder
|   └── second.txt
:
└── last_folder
    └── last.txt

where first.txt-last.txt will contain paths to specific files that contain string /Jan/2016.

first.txt
first_folder/11.11/access_log.9.gz
first_folder/11.11/access_log.8.gz
first_folder/11.12/access_log.9.gz
first_folder/11.12/access_log.8.gz

Question #4 (22.9.2016)

I needed to modify /Jan/2016 to certain period of time...for example 1/Nov/2014-31/Apr/2015 so in a code that @Zana provided instead of the /Jan/2016 I used /(Nov|Dec)/2014|/(Jan|Feb|Mar|Apr)/2015. Warning is thrown:

xargs: Warning: a NUL character occurred in the input.  It cannot be passed through in the argument list.  Did you mean to use the --null option?

All results are not returned in created files, althought all files are created.

3
  1. Have data that contains string in a separate text files depending on a folder (example: first folder - first.txt etc)

You could use a very simple loop to make a text file with the records for that directory inside the directory:

for d in ~/Desktop/logs/* ; do zgrep -E '/Jan/2016' "$d"/* >"$d"/out.txt ; done

Desktop/logs
├── first_folder
│   ├── access_log
│   ├── access_log.gz
│   └── out.txt
└── second_folder
    ├── access_log
    ├── access_log.gz
    └── out.txt

Using this slight adjustment to waltinator's script:

for d in ~/Desktop/logs/* ; do
    if [[ -d "$d" ]] ; then
        outname="$d".txt
        find "$d" -type f -print0 | xargs -0 zgrep -E '/Jan/2016' >"$outname"
    fi
done

will give this structure:

├── first_folder
│   ├── access_log
│   └── access_log.gz
├── first_folder.txt
├── second_folder
│   ├── access_log
│   └── access_log.gz
└── second_folder.txt

Notes

  • for d in ~/Desktop/logs/* ; do loop over the contents of ~/Desktop/logs and do something with them
  • if [[ -d "$d" ]]; then do something only if $d is a directory
  • find "$d" -type f -print0 search inside directories $d for files and output them with the null separator so that we can use
  • xargs -0 construct a command with the output of the previous command as arguments, using the null character as separator (otherwise filenames with spaces break this approach)
  • zgrep searches in zipped/compressed archives
  • First loop gives me: bash: ~Desktop/logs/*/out.txt: No such file or directory. Second one works like a charm! Thanks so much! – vayacondios2015 Sep 14 '16 at 19:23
  • I have put this as a correct answer because this is what I was looking for. This gives an answer to the question #2! – vayacondios2015 Sep 14 '16 at 19:24
  • @vayacondios2015 oops sorry I just missed a / from the loop - now fixed. Glad the second way worked how you wanted :) – Zanna Sep 14 '16 at 20:03
4

The modern way to use find, bearing in mind that you will eventually see a filename containing spaces, is with -print0 and xargs -0:

# list all filenames containing '/Jan/2016'
find ~/Desktop/logs -type f -print0 | xargs -0 zgrep -l '/Jan/2016'
# 1. Have all the data from all folders that contain that string under one text file
find ~/Desktop/logs -type f -print0 | xargs -0 zgrep -l '/Jan/2016' >one.text.file
#
# 2. Have data that contains string in a separate text files depending on a folder (example: first folder - first.txt etc)
pushd ~/Desktop/logs
for dir in * ; do
    if [[ -d "$dir" ]] ; then
        outname="$dir.txt"
        find "~/Desktop/logs/$dir" -type f -print0 | xargs -0 zgrep -l '/Jan/2016' >"$outname"
    fi
done
popd
  • Thanks for quick respond! Under 1. this gives me a list of folders that contain that string, but I would like to get actual data (all the logs). Under 2. find: `~/Desktop/logs/first_folder': No such file or directory...it seems to me that it doesnt search for and under 11.11 etc folders. – vayacondios2015 Sep 13 '16 at 20:16
  • I have edited my question regardless the issue under 1. – vayacondios2015 Sep 13 '16 at 20:51
  • 2
    ah it's just a little slip there, instead of find "~/Desktop/logs/"$dir" it should be find "$dir" otherwise you get something like /home/$USER/Desktop/logs/home/$USER/Desktop/logs/first_folder, that's the only reason it didn't work @vayacondios2015 otherwise it's very nice – Zanna Sep 14 '16 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.