3

Am new (2 days old) to Linux and grep and am stuck here. Scenario. I have data lasting more than 10yrs which I have been doing manually until I came across grep. The folders are of form /yyyy/mm/dd i.e. day1, day2 till end of month. I need to search for a specific string iteration 8. If found, then I need to copy the previous 3 rows from row where iteration 8 is located. Then I need to transpose the into an output file. This is how am attempting to achieve my dilemma. Since am unable to transpose am trying to split the outputs then combine later. Please guide me on this case.

 for file in /filepath/snc* #adding full path
     do
      echo $file
       grep -r " Mean" $file | awk '{print $1 " " $2}'> mean.txt # to enable single columns for ease of manipulation later
       grep -r " RMS" $file | awk '{print $1 " " $2}' > rms.txt
       grep -r " o-c" $file | awk '{print $3 " "$4}' > o-c.txt
       grep -rl "iteration 8" $file > iteration.txt # to verify that the files are the correct ones
      done

paste iteration.txt o-c.txt mean.txt rms.txt > daily-summary.txt #the output file must be in this specific order
grep "iteration 8" daily-summary.txt | awk '{print $3 " " $4 " " $5 " " $6 " " $7 " " $8}' >> monthly-summary-path.txt

#grep -3 "iteration 8" daily-summary.txt  >> monthly-summary-file.txt # two lines before

rm mean.txt rms.txt std.txt

Sample input file:

            Mean    -78.6
            rms      1615
            o-c      1612.97456

iteration 8

Sample output file:

year month day o-c         mean  rms
2015   12   12  1612.97456 -78.6 1615
2015   12   11  1525.36589 -78.0 1642

=======================   
14
  • Let's say you have two iteration 8 strings in day1 and one iteration 8 string in day2. Do the target 6 lines of the first file and the target 3 lines of the second file go in different files? Do all 9 target lines go toghether in a single file? Else?
    – kos
    Dec 22, 2015 at 10:10
  • 1
    -B is the option of grep which allows you to print previous lines to a matched pattern. So grep -B3 "iteration8" ... will be a good start, though I'm sure you'll get some well crafted answers to help you achieve your entire goal. Also I think there's a typo in your question, as you switch from iteration8 to iteration18.
    – Arronical
    Dec 22, 2015 at 10:13
  • @kos Thanks. Ok. There can only one instance of iteration 8 or nothing at all per day. Dec 22, 2015 at 10:14
  • 1
    It's a bit hard to understand what you're after without looking at a sample input / sample output. Can you post a short sample input file and a sample of how the output file for that input file should look like?
    – kos
    Dec 22, 2015 at 10:28
  • 1
    @user3192045 you will need to provide the link URL for your pastebin - please edit it into your question instead of adding it in a comment Dec 22, 2015 at 13:25

1 Answer 1

1

This will create a report for a single month:

#!/usr/bin/perl

use strict;
use warnings;

@ARGV == 1 || die($!);

my $realpath = `realpath $ARGV[0]`;
chomp($realpath);

opendir(my $dir, $realpath) || die($!);

my @files;

while(readdir($dir)) {
    -f "$realpath/$_" && push(@files, "$realpath/$_");
}

print("year\tmonth\tday\to-c\tmean\trms\n");

my @realpath_s = split("/", $realpath);

foreach my $file (sort(@files)) {
    open(my $in, $file) || die($!);

    while(<$in>) {
        if(/^\s*Mean/) {
            my @row;
            for(my $i = 0; $i < 3; $i++) {
                my @F = split(/\s/);
                push(@row, $F[2]);
                $_ = <$in>;
            }
            $_ = <$in>;
            my @F = split(/\s/);
            if($F[1] == 8) {
                $file =~ s/.*day//;
                print("$realpath_s[@realpath_s-2]\t$realpath_s[@realpath_s-1]\t$file\t$row[2]\t$row[0]\t$row[1]\n");
                last;
            }
        }
    }
}

print("\n=======================\n");

exit 0;

Save it to, say, ~/script.pl, and call it passing the path to the reports of a month:

perl ~/script.pl /path/to/2015/12

The output will be printed to the the terminal; you can use a redirection to redirect it to a file:

perl ~/script.pl /path/to/2015/12 > ~/report_2015_12.txt

It should be fairly easy to script multiple calls in a Bash script to create yearly / 10-year reports.

% tree
.
├── 2015
│   └── 12
│       ├── day1
│       ├── day2
│       └── day3
└── script.pl

2 directories, 4 files
% perl script.pl 2015/12
year    month   day o-c mean    rms
2015    12  1   1612.97456  -78.6   1615
2015    12  2   1612.97456  -79.6   1615
2015    12  3   1612.97456  -80.6   1615

=======================

In the example all files in 2015/12 contain a iteration 8 line, hence a line is printed for each of them.

1
  • Thank you. Will have a look soon and revert. The output looks exactly as intended. @steeldriver, thanks for your input too. Dec 23, 2015 at 11:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .