Specific edits to a text file (awk?)

Question

I have a file that looks like

TITLE
1.000000000000000
10.0000000000000000    0.0000000000000000    0.0000000000000000 
0.0000000000000000   10.0000000000000000    0.0000000000000000
0.0000000000000000    0.0000000000000000   10.0000000000000000
U   U
X   X
C
0.2000000000000028  0.2000000000000028  0.2000000000000028
0.2967599999999990  0.0641000000000034  0.1551499999999990
0.1033699999999982  0.3361099999999979  0.244990000000001

and I need to have a script that will modify the bottom number block (below C to 30 less than their original values. Is there any such way to do so?

So far, the best I've got is

$ awk '{if(NR>1){for(i=2;i<=NF;i++){$(i)=$(i)-10;}}print;}' data.txt | column -t

but, that's from the internet and I'm not sure how to manipulate it myself to the desired effect. This however will not print/overwrite what's currently in data.txt, which is what I'm going for.

Thanks for the help!

Answer 1

Given data.awk below:

{
        if (matched) {
                for (i = 1; i <= NF; i++) {
                        $(i) = 30.0 - $(i)
                }
        }
        print
}
/^C/ { matched = 1 }
BEGIN { CONVFMT = "%.20f" }

You get:

$ awk -f data.awk data.txt
TITLE
1.000000000000000
10.0000000000000000    0.0000000000000000    0.0000000000000000 
0.0000000000000000   10.0000000000000000    0.0000000000000000
0.0000000000000000    0.0000000000000000   10.0000000000000000
U   U
X   X
C
29.79999999999999715783 29.79999999999999715783 29.79999999999999715783
29.70324000000000097543 29.93589999999999662350 29.84485000000000098908
29.89663000000000181444 29.66389000000000208956 29.75500999999999862666

It obviously has precision problem for your input. So you might want to call bc command for actual calculations (it supports arbitrary precision):

{
        if (matched) {
                for (i = 1; i <= NF; i++) {
                        cmd = "echo 30.0 - " $(i) " | bc"
                        cmd | getline $(i)
                        close(cmd)
                }
        }
        print
}
/^C/ { matched = 1 }

Result:

TITLE
1.000000000000000
10.0000000000000000    0.0000000000000000    0.0000000000000000 
0.0000000000000000   10.0000000000000000    0.0000000000000000
0.0000000000000000    0.0000000000000000   10.0000000000000000
U   U
X   X
C
29.7999999999999972 29.7999999999999972 29.7999999999999972
29.7032400000000010 29.9358999999999966 29.8448500000000010
29.8966300000000018 29.6638900000000021 29.755009999999999

To overwrite data.txt with the result, you normally need to write it to another file, then rename it to the original file.

$ awk -f data.awk data.txt > data.txt.out
$ mv data.txt.out data.txt

Or use sponge in moreutils.

$ sudo apt-get install moreutils
$ awk -f data.awk data.txt | sponge data.txt

Answer 2

Here's my awk version :

awk '/^C/,0 {for (i=1;i<=NF;i++) { if ( $i != "C" ) printf "%.16f ",$i-30.0000};print"\n" }' data.txt

Here we get all the stuff from C character till the end of file, subtract 30 in each column, add newline, and repeat the process.The if statement added to avoid subtracting 30 from C, obviously.

The output is this :

    46)serg@ubuntu[/home/xieerqi]
    >_ awk '/^C/,0 {for (i=1;i<=NF;i++) { if ( $i != "C" )  printf "%.16f ",$i-30.0000};print"\n" }' data.txt                             

-29.7999999999999972 -29.7999999999999972 -29.7999999999999972 

-29.7032400000000010 -29.9358999999999966 -29.8448500000000010 

-29.8966300000000018 -29.6638900000000021 -29.7550099999999986 

This can be substituted in the original file; Alternativelly we could always try to print stuff before C with a BEGIN { } statement

Answer 3

Using python:

#!/usr/bin/env python2
import decimal
with open('/path/to/data.txt') as f:
    for line in f:
        if line.rstrip() == 'C':
            print line.rstrip()
            break
        else:
            print line.rstrip()
    for line in f:
        print '\t'.join(['{0:.16f}'.format(decimal.Decimal(30 - float(part))) for part in line.rstrip().split()])

Output :

TITLE
1.000000000000000
10.0000000000000000    0.0000000000000000    0.0000000000000000
0.0000000000000000   10.0000000000000000    0.0000000000000000
0.0000000000000000    0.0000000000000000   10.0000000000000000
U   U
X   X
C
29.7999999999999972 29.7999999999999972 29.7999999999999972
29.7032400000000010 29.9358999999999966 29.8448500000000010
29.8966300000000018 29.6638900000000021 29.7550099999999986

• Every time python reads a line of the file the pointer is increased by one to point to the next line, we are utilizing this to read and print upto the line that contains only C.

• For the lines after C we have splitted the line into parts by line.rstrip().split() and then subtracted each part from 30 to get the desired result.

• To get precision on the resultant floating point number we have used the decimal module.

Answer 4

I can't provide a good solution to the problem, but I'll try to describe in depth what the problem is and I'll provide a partial solution.

The problem:

Floating point numbers on machines suffer from limited precision: in short, only a limited subset of floating point numbers [per each order of magnitude] is representable.

Floating point numbers on machines are represented closely following the normalized notation ± significand * base ^ exponent (where base = base of representation, significand = any real number > 0 and <= the base of representation and where exponent = order of magnitude): for example, on a 32-bit machine following the IEEE 754 standard, single-precision floating point numbers are represented using the first bit to represent the sign, the following 8 bits to represent the order of magnitude and the last 23 bits to represent the significand, while double-precision floating point numbers are represented using the first bit to represent the sign, the following 11 bits to represent the order of magnitude and the last 52 bits to represent the significand (the base, being always 2, is not represented). For this, a number's significand has to be represented always using 23 bits (using single-precision) or using 52 bits (using double-precision).

A property of this way of representing floating point numbers on a fixed number of bits is that being the number of representable significands per order of magnitude always the same, the average "distance" between representable floating point numbers with the same order of magnitude increases as the order of magnitude of the two increases.

For the above, the first problem is that if a floating point number's normalized notation's significand is not in the limited set of representable significands, it's rounded to the closest (higher or lower) significand representable.

Speaking of numbers represented with the same order of magnitude, a second problem is that even when a floating point number is representable precisely, adding / substracting another [precisely representable] floating point number to it might result in a not precisely representable floating point number, whose significand will be rounded to the closest (higher or lower) significand representable.

Finally, speaking of numbers represented with a different order of magnitude, the third problem (mostly due to the CPU architecture) is that in order to be able to perform additions / substractions between floating point numbers represented with a different order of magnitude, the numbers need to be first represented using the same order of magnitude; this implies that the smallest one's order of magnitude needs to be increased, and that (in order to balance this) its significand needs to be shifted to the right, with the consequent loss of the number of bits exceeding the 23 / 52 available; if this is not enough, floating point numbers with a significant difference in their order of magnitude might result, once added / substracted, exactly in the number with the highest absolute value, this for the already stated problem (not enough difference to step the non representable significand up / down to a different higher / lower representable significand) and increasingly worse as the order of magnitude of two numbers diverges further.

The implications of all of this are: you'll never be sure to get an accurate result using floating point math, however this can be mitigated by using an higher-precision representation.

The partial solution:

For the above, the results of these awk one-liners are not precise; this could have been mitigated by the usage of double-precision in their printf commands, but this is not supported.

---

This will decrease by 30 the value of the first 3 space-separated numbers in each line after the first line matching C, keeping the numbers' format. Since the awk version included in Ubuntu does not support in-place edits, you'll have to either use awk and redirect its stdout to a file using bash's > operator or use gawk (GNU awk) >= 4.10.0;

Using awk:

awk 'NR==1, $0=="C"; $0=="C", 0 {if ($0!="C") printf "%.16f  %.16f  %.16f\n", $1-30, $2-30, $3-30}' data.txt > data_processed.txt

Using gawk (GNU awk) >= 4.10.0

gawk -i inplace 'NR==1, $0=="C"; $0=="C", 0 {if ($0!="C") printf "%.16f  %.16f  %.16f\n", $1-30, $2-30, $3-30}' data.txt

• NR==1, $0=="C";: selects and prints all the records between the first and the first matching C inclusive;
• $0=="C", 0 {if ($0!="C") printf "%.16f %.16f %.16f\n", $1-30, $2-30, $3-30}: selects all the records between the first matching C and the last inclusive and prints the 1st, 2nd and 3rd field of each selected record not matching C double-space separated and decreased by 30 keeping the original number's format;

Sample output:

~/tmp$ cat data.txt
TITLE
1.000000000000000
10.0000000000000000    0.0000000000000000    0.0000000000000000 
0.0000000000000000   10.0000000000000000    0.0000000000000000
0.0000000000000000    0.0000000000000000   10.0000000000000000
U   U
X   X
C
0.2000000000000028  0.2000000000000028  0.2000000000000028
0.2967599999999990  0.0641000000000034  0.1551499999999990
0.1033699999999982  0.3361099999999979  0.244990000000001
~/tmp$ awk 'NR==1, $0=="C"; $0=="C", 0 {if ($0!="C") printf "%.16f  %.16f  %.16f\n", $1-30, $2-30, $3-30}' data.txt
TITLE
1.000000000000000
10.0000000000000000    0.0000000000000000    0.0000000000000000 
0.0000000000000000   10.0000000000000000    0.0000000000000000
0.0000000000000000    0.0000000000000000   10.0000000000000000
U   U
X   X
C
-29.7999999999999972  -29.7999999999999972  -29.7999999999999972
-29.7032400000000010  -29.9358999999999966  -29.8448500000000010
-29.8966300000000018  -29.6638900000000021  -29.7550099999999986