Oneliners by me and other persons as well as some scripts tested
If the order of the items and the separators can be different from what you specify in the question, I thought the following one-liner would do it,
< input tr ' ' '|' | cut -d '|' -f 4,6,10 > output
but in a comment you wrote that you need exactly the specified format.
I added a solution with 'awk', which is approximately on par with PerlDuck's solution with perl
. See the end of this answer.
< input awk '{gsub("\\|"," "); print $5 ":" $9 "," $3}' > output
Test of oneliners and small scripts
The test was done in my computer with Lubuntu 18.04.1 LTS, 2*2 processors and 4 GiB RAM.
I made a huge infile
by 'doubling 20 times' from your demo input
(1572864 lines), so some margin to your 500000 lines,
Oneliner with cut
and sed
:
$ < infile cut -d '|' -f 3,5,6 | sed -e 's/|[A-Z].*, /|/' -e 's/ )$//' > outfile
$ wc -l infile
1572864 infile
$ wc -l outfile
1572864 outfile
$ tail outfile
12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1
12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1
12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1
12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1
Timing
We might expect, that a pure sed
solution would be faster, but I think that reordering of the data slows it down, so that the cut
and sed
solution is faster. Both solutions work without any problem in my computer.
Oneliner with cut
and sed
:
$ time < infile cut -d '|' -f 3,5,6 | sed -e 's/|[A-Z].*, /|/' -e 's/ )$//' > outfile
real 0m8,132s
user 0m8,633s
sys 0m0,617s
A pure sed
oneliner by xenoid:
$ time sed -r 's/^[^|]+\|[^|]+\|([^|]+)\|[^|]+\|([^|]+)\|.+\( .+, ([^ ]+).+/\2:\3,\1/' <infile > outfile-sed
real 1m8,686s
user 1m8,259s
sys 0m0,344s
A python
script using a regex with non-greedy matches by xeniod:
#!/usr/bin/python
import sys,re
pattern=re.compile(r'^[^|]+?\|[^|]+?\|([^|]+?)\|[^|]+?\|([^|]+?)\|[^,]+?, (.+) \)\|$')
for line in sys.stdin:
match=pattern.match(line)
if match:
print(match.group(2)+':'+match.group(3)+','+match.group(1))
$ time < infile ./python-ng > outfile.pyng
real 0m8,055s
user 0m7,359s
sys 0m0,300s
$ python --version
Python 2.7.15rc1
A perl
oneliner by PerlDuck is faster than the previous oneliners:
$ time perl -lne 'print "$2:$3,$1" if /^(?:[^|]+\|){2}([^|]+)\|[^|]+\|([^|]+)\|[^,]+,\s*(\S+)/;' < infile > outfile.perl
real 0m5,929s
user 0m5,339s
sys 0m0,256s
Oneliner with tr
and cut
with a tr -s
command:
I used tr
to convert the spaces in the input file to pipeline characters and then cut
could do it all without sed
. As you can see, tr
is much faster than sed
. The tr -s
command removes double pipes in the input, which is a good idea, particularly if there can be repeated spaces or pipes in the input file. It does not cost much.
$ time < infile tr ' ' '|' | tr -s '|' '|' | cut -d '|' -f 3,5,9 > outfile-tr-cut
real 0m1,277s
user 0m1,781s
sys 0m0,925s
Oneliner with tr
and cut
without the tr -s
command, fastest so far:
time < infile tr ' ' '|' | cut -d '|' -f 4,6,10 > outfile-tr-cut
real 0m1,199s
user 0m1,020s
sys 0m0,618s
$ tail outfile-tr-cut
12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1
12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1
12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1
12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1
12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1
Oneliner with awk
, fast but not the fastest,
< input awk '{gsub("\\|"," "); print $5 ":" $9 "," $3}' > output
$ time < infile awk '{gsub("\\|"," "); print $5 ":" $9 "," $3}' > outfile.awk
real 0m5,091s
user 0m4,724s
sys 0m0,365s
awk
with parallel
implemented according to Ole Tange reduces the real time from 5s to 2s:
#!/bin/bash
doit() {
awk '{gsub("\\|"," "); print $5 ":" $9 "," $3}'
}
export -f doit
parallel -k --pipepart --block -1 -a infile doit > outfile.parallel-awk
$ time ./parallel-awk
# Academic tradition requires you to cite works you base your article on.
# When using programs that use GNU Parallel to process data for publication
#please cite:
# O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
# ;login: The USENIX Magazine, February 2011:42-47.
# This helps funding further development; AND IT WON'T COST YOU A CENT.
#If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
# To silence this citation notice: run 'parallel --citation'.
real 0m1,994s
user 0m5,015s
sys 0m0,984s
We can expect that the advantage with parallel
will increase with bigger size of the input file as described by the diagram in Ola Tange's answer to this question.
Speed summary: the 'real' time according to time
rounded to 1 decimal
1m 8.7s - sed
8.1s - cut & sed
7.4s - python
5.9s - perl
5.1s - awk
2.0s - parallel & awk
1.2s - tr & cut
Finally, I note that the oneliners with sed
, python
, perl
, awk
and {parallel
& awk
} create an output file with the prescribed format.
$ tail outfile.awk
202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT75.SERV1,12-664-1186
202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.SERV1,12-654-0330
202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.C1.P1,12-654-0330
202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT75.SERV1,12-664-1186
202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.SERV1,12-654-0330
202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.C1.P1,12-654-0330
202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT75.SERV1,12-664-1186
202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.SERV1,12-654-0330
202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.C1.P1,12-654-0330
202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT75.SERV1,12-664-1186
wait
after some number of lines, or investigate other strategies to parallelize the job (such as GNU parallel) – glenn jackman Jan 31 '19 at 19:34sed
instruction operating on the input file that would run thousands of times faster.awk
would also be a solution. – xenoid Feb 1 '19 at 8:40