1

I have a .data and I need to re-organise it so that 3 consecutive lines are joined together in a single line. Just to be clear, I have the following ...

D611102 = 'SVM_PRS_Hydr_L01', T = 0.0,
C = 3.341441E-006 * Cp_SVM_PRS_Pi001 * Dens_SVM_PRS_Pi001,
A = 0.007425, ALP = 1.000000, EPS = 1.000000,
FX = -0.355305, FY = 0.857782, FZ = 0.282590;

...and so on

and I need all that in a single line:

D611102 = 'SVM_PRS_Hydr_L01', T = 0.0,C = 3.341441E-006 * Cp_SVM_PRS_Pi001 ...

and I need it through all the document. Is there a fast way to do so?

The other difficulty is that all the info to be joined are sometimes in 3 different lines and sometimes in 2 or 4 different lines. The only thing I have is that each line is identified by a D.... Is it possible to do it? How?

Basically each new line should start with D611102 that is the node number. And I need a single big line with T = ..., C = ..., till FZ = .... ,etc. Each full line must have all the data till the ;

To be more clear, I have the following: ...

   D611102 = 'SVM_PRS_Hydr_L01', T = 0.0,
     C = 3.341441E-006 * Cp_SVM_PRS_Pi001 * Dens_SVM_PRS_Pi001,
     A = 0.007425, ALP = 1.000000, EPS = 1.000000,
     FX = -0.355305, FY = 0.857782, FZ = 0.282590;
    D611103 = 'SVM_PRS_Hydr_L01', T = 0.0,
     C = 3.341441E-006 * Cp_SVM_PRS_Pi001 * Dens_SVM_PRS_Pi001,
     A = 0.007425, ALP = 1.000000, EPS = 1.000000,
     FX = -0.656518, FY = 0.656518, FZ = 0.282590;

... etc ... And I need this:

D611102 = 'SVM_PRS_Hydr_L01', T = 0.0,C = 3.341441E-006 * Cp_SVM_PRS_Pi001 * Dens_SVM_PRS_Pi001,A = 0.007425, ALP = 1.000000, EPS = 1.000000,FX = -0.355305, FY = 0.857782, FZ = 0.282590; (all in a single line)
D611103 = 'SVM_PRS_Hydr_L01', T = 0.0,C = 3.341441E-006 * Cp_SVM_PRS_Pi001 * Dens_SVM_PRS_Pi001,A = 0.007425, ALP = 1.000000, EPS = 1.000000,FX = -0.656518, FY = 0.656518, FZ = 0.282590; 

(all in a single line) and so on for the whole document.

3
  • Please edit and clarify. Do you want all the data of the file one one, single, long line? Do you want every occurrence of D to be on a different line? What about the Dens_SVM_PRS_Pi001 in your example? Or is it that lines are defined as starting with a D and ending with a ;? If so, can we be sure that there are no ; in between?
    – terdon
    Dec 9, 2015 at 10:05
  • Ok, basically each new line should start with D611102 that is the node number. And I need a single big line with T = ..., C = ..., till FZ = .... and yes, each full line must have all the info till the ;
    – sarde88
    Dec 9, 2015 at 10:09
  • Hi sande88 did you have the chance to try the options? Dec 11, 2015 at 7:39

2 Answers 2

2

If I understand correctly, you basically want to remove all line breaks that don't occur directly after a ;. If so, you can do:

perl -pe 's/(?<!;)\s*\n/ /' file > newfile

Or, to edit the file in place, use -i:

perl -i.bak -pe 's/(?<!;)\s*\n/ /' file

The above will make the changes to file and create a backup of the original called file.bak. To skip creating the backup just use -i alone, without an extension.

Explanation

The -p means "print every input line after applying the script given by -e. The s/// is the substitution operator. Its general format is s/pattern/replacement/ and will replace pattern with replacement.

In this case, pattern is 0 or more whitespace characters (\s*) followed by a newline (\n) which are not preceded by a ;. The (?<!foo)bar construct is a negative lookbehind, it will match bar if the previous characters are not foo. Therefore, the script above will remove all newlines that aren't just after a ;.

4
  • Ok thanks a lot. I think your answer solves my problem! Right now I don't have my pc with ubuntu with me, but this evening i'll try your solution. Is there another way for doing so also in windows? With notepad++ or something similar?
    – sarde88
    Dec 9, 2015 at 10:29
  • @sarde88 no idea, I don't use Windows. I think that notepad++ has good regex support though, you might try using the regex there. However, note that if you open the file in Windows, you will change the line endings to \r\n instead of \n. You don't want to do that. If you find that it doesn't work after having opened the file on Windows, use this instead: perl -pe 's/(?<!;)\s*\r*\n/ /' file > newfile.
    – terdon
    Dec 9, 2015 at 10:33
  • ok thanks, this evening I'll try. One tipp: how do you get "proficient" in using the linux terminal? I've been using it for like 2 years, but I am only familiar with basic commands (change directory, create folders, move file, ...). For the rest, unfortunately, I look for the code in internet and use it...but I don't see any improvement in my skills. Any useful book?Just practice and practice? Thanks again.
    – sarde88
    Dec 9, 2015 at 10:40
  • @sarde88 actually using it (if you only know those simple commands, you're not really using it) and being curious. For every command you run, read man command to learn how it works. Read a basic scripting tutorial. It also helps to have been using Linux at home and work for ~17 years or so :)
    – terdon
    Dec 9, 2015 at 10:48
1

The script below should do the job. Since it reads per line, it should be relatively fast on larger files, but didn't test it on a large file.

#!/usr/bin/env python3
import sys

f = sys.argv[1]

s = ""

with open(f) as lines:
    for l in  lines:
        if l.startswith("D"):
            print(s+l.strip(), end = "")
            s = "\n"
        else:
            print(l.strip(), end = "")

Use it

  • Copy it into an empty file, save it as combine_lines.py
  • Run it by the command:

    python3 /path/to/combine_lines.py <.data_file>
    

Explanation

The script reads the lines, line by line. If the line starts with a D, else the line is simply printed after the existing line, except for the first line.

Test:

D611102 = 'SVM_PRS_Hydr_L01', T = 0.0,C = 3.341441E-006 * Cp_SVM_PRS_Pi001 * Dens_SVM_PRS_Pi001,
A = 0.007425, ALP = 1.000000, EPS = 1.000000,
FX = -0.355305, FY = 0.857782, FZ = 0.282590;
D611102 = 'SVM_PRS_Hydr_L01', T = 0.0,C = 3.341441E-006 * Cp_SVM_PRS_Pi001 * Dens_SVM_PRS_Pi001,
A = 0.007425, ALP = 1.000000, EPS = 1.000000,
FX = -0.355305, FY = 0.857782, FZ = 0.282590;

becomes:

D611102 = 'SVM_PRS_Hydr_L01', T = 0.0,C = 3.341441E-006 * Cp_SVM_PRS_Pi001 * Dens_SVM_PRS_Pi001,A = 0.007425, ALP = 1.000000, EPS = 1.000000,FX = -0.355305, FY = 0.857782, FZ = 0.282590;
D611102 = 'SVM_PRS_Hydr_L01', T = 0.0,C = 3.341441E-006 * Cp_SVM_PRS_Pi001 * Dens_SVM_PRS_Pi001,A = 0.007425, ALP = 1.000000, EPS = 1.000000,FX = -0.355305, FY = 0.857782, FZ = 0.282590;

EDIT

or,

as suggested by @terdon, using the ending ";" as a trigger, which gives us the opportunity to skip the s = trick:

#!/usr/bin/env python3
import sys

f = sys.argv[1]

with open(f) as lines:
    for l in  lines:
        l = l if l.endswith(";\n") else l.strip(); print(l, end = "")

Comparing Perl to python

On a relatively large file, 550MB, 9006121 lines:

Perl:

$ time perl -pe 's/(?<!;)\s*\n/ /' '/home/jacob/Bureaublad/data_large' > '/home/jacob/Bureaublad/data_large2'

real    0m27.171s
user    0m25.536s
sys     0m1.054s

Python:

time '/home/jacob/Bureaublad/pscript_9.py' '/home/jacob/Bureaublad/data_large' > '/home/jacob/Bureaublad/data_large2'

real    0m15.235s
user    0m13.806s
sys     0m1.279s

On a smaller file, 51KB, 838 lines:

$ time perl -pe 's/(?<!;)\s*\n/ /' '/home/jacob/Bureaublad/data_small' > '/home/jacob/Bureaublad/data_small2' 

real    0m0.008s
user    0m0.007s
sys     0m0.000s

Python:

$ time '/home/jacob/Bureaublad/pscript_9.py' '/home/jacob/Bureaublad/data_small' > '/home/jacob/Bureaublad/data_small2' 

real    0m0.033s
user    0m0.019s
sys     0m0.011s

The bottom line is that if you have bigger files, python might be what you'd like to use, if you have many, smaller files, Perl is the better option.

2
  • 1
    You might want to use endswith(";") (or whatever the pythonspeak for that is) instead of startwith("D"). The OP is not ver clear but I think that the only thing we can be sure of is that lines start with D and end with ;. There is no guarantee that one of the bad line breaks won't have been inserted after a D.
    – terdon
    Dec 9, 2015 at 10:51
  • @terdon of course, much smarter, since it skips the s= section. Will edit in an hour. Dec 9, 2015 at 11:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .