12

Currently, I have multiple text files with contents looking like this (with many lines):

565 0 10 12 23 18 17 25
564 1 7 12 13 16 18 40 29 15

I wish to change each line to have the following format:

0 565:10:1 565:12:1 565:23:1 565:18:1 565:17:1 565:25:1
1 564:7:1 564:12:1 564:13:1 564:16:1 564:18:1 564:40:1 564:29:1 564:15:1

Are there any way of doing the above using sed? Or do I need to resort to Python?

22

You could do it with sed, yes, but other tools are simpler. For example:

$ awk '{
        printf "%s ", $2; 
        for(i=3;i<=NF;i++){
            printf "%s:%s:1 ",$1,$(i) 
        }
        print ""
       }' file 
0 565:10:1 565:12:1 565:23:1 565:18:1 565:17:1 565:25:1 
1 564:7:1 564:12:1 564:13:1 564:16:1 564:18:1 564:40:1 564:29:1 564:15:1 

Explanation

awk will split each line of input on whitespace (by default), saving each fields as $1, $2, $N. So:

  • printf "%s ", $2; will print the 2nd field and a trailing space.
  • for(i=3;i<=NF;i++){ printf "%s:%s:1 ",$1,$(i) } : will iterate over fields 3 to the last field (NF is the number of fields) and for each of them it will print the 1st field, a :, then the current field and a :1.
  • print "" : this just prints a final newline.

Or Perl:

$ perl -ane 'print "$F[1] "; print "$F[0]:$_:1 " for @F[2..$#F]; print "\n"' file 
0 565:10:1 565:12:1 565:23:1 565:18:1 565:17:1 565:25:1 
1 564:7:1 564:12:1 564:13:1 564:16:1 564:18:1 564:40:1 564:29:1 564:15:1 

Explanation

The -a makes perl behave like awk and split its input on whitespace. Here, the fields are stored in the array @F, meaning that the 1st field will be $F[0], the 2nd $F[1] etc. So:

  • print "$F[1] " : print the 2nd field.
  • print "$F[0]:$_:1 " for @F[2..$#F]; : iterate over fields 3 to the last field ($#F is the number of elements in the array @F, so @F[2..$#F] takes an array slice starting at the 3rd element until the end of the array) and print the 1st field, a :, then the current field and a :1.
  • print "\n" : this just prints a final newline.
12

Here is a horrible sed way!

$ sed -r 's/^([0-9]+) ([0-9]+) ([0-9]+)/\2 \1:\3:1/; :a s/([0-9]+)(:[0-9]+:1) ([0-9]+)( |$)/\1\2 \1:\3:1 /; t a; s/ $//' file
0 565:10:1 565:12:1 565:23:1 565:18:1 565:17:1 565:25:1 
1 564:7:1 564:12:1 564:13:1 564:16:1 564:18:1 564:40:1 564:29:1 564:15:1

More readably:

sed -r '
s/^([0-9]+) ([0-9]+) ([0-9]+)/\2 \1:\3:1/
:a 
s/([0-9]+)(:[0-9]+:1) ([0-9]+)( |$)/\1\2 \1:\3:1 /
t a
s/ $//'

Notes

  • -r use ERE
  • s/old/new/ replace old with new
  • ^([0-9]+) save some numbers at the start of the line
  • \1 backreference to first saved pattern
  • :a label this section of the script a
  • ( |$) either a space or the end of the line
  • t test whether the last replacement was successful - if it was, then do the next command
  • a find the label :a and do it again
  • s/ $// remove the trailing space

So after adding the structure to the first part, we repeatedly find the last instance of the structure and apply it to the next number...

But I agree other tools make it easier...

  • I was waiting for your sed solution :D – Ravexina Jun 5 '17 at 8:57
  • :D it took me a while @Ravexina - I reckon muru can make a cleaner one – Zanna Jun 5 '17 at 8:59
5

With awk:

awk '{printf "%s ",$2; for (i=3; i<=NF; i++) printf $1":"$i":1 "; printf "\n"}' file

or with bash:

while read -r -a a; do                  # read line to array a
  printf "%s " ${a[1]}                  # print column #1
  for ((i=2;i<${#a[@]};i++)); do        # loop from column #2 to number of columns
    printf "%s " "${a[0]}:${a[$i]}:1"   # print content/values
  done
  echo                                  # print line break
done < file                             # read file from stdin

Output:

0 565:10:1 565:12:1 565:23:1 565:18:1 565:17:1 565:25:1 
1 564:7:1 564:12:1 564:13:1 564:16:1 564:18:1 564:40:1 564:29:1 564:15:1 
5

Well, you can do it in sed, but python works also.

$ ./reformatfile.py  input.txt                                                                        
0 565:10:1 565:12:1 565:23:1 565:18:1 565:17:1 565:25:1
1 564:7:1 564:12:1 564:13:1 564:16:1 564:18:1 564:40:1 564:29:1 564:15:1

Contents of the reformatfile.py are as so:

#!/usr/bin/env python3
import sys

with open(sys.argv[1]) as fd:
    for line in fd:
        words = line.strip().split()
        pref = words[0]
        print(words[1],end=" ")
        new_words = [ ":".join([pref,i,"1"]) for i in words[2:] ]
        print(" ".join(new_words))

How does this work? There's really nothing particularly special going on. We open first command-line argument as file for reading and proceed with breaking down each line into "words" or individual items. First words becomes pref variable, and we print on stdout second ( words[1] ) item ending with space. Next we construct new set of "words" via list comprehensions and .join() function on a temporary list of pref, each word, and string "1". Final step is to print those out

4

With awk:

awk '{printf("%s ", $2); for(i=3; i<NF; i++) printf("%s:%s:1 ", $1, $i);\
          printf("%s:%s:1\n", $1, $NF)}' file.txt

It's all about formatting space-separated fields in desired format:

  • printf("%s ", $2) prints the second field with a trailing space

  • for(i=3; i<NF; i++) printf("%s:%s:1 ", $1, $i) iterates over the 3rd to second last fields and prints the fields in desired format (first field, then a colon, then the current field, then a colon, finally 1) with a trailing space

  • printf("%s:%s:1\n", $1, $NF) prints the last field with newline

Example:

% cat file.txt
565 0 10 12 23 18 17 25
564 1 7 12 13 16 18 40 29 15

% awk '{printf("%s ", $2); for(i=3; i<NF; i++) printf("%s:%s:1 ", $1, $i); printf("%s:%s:1\n", $1, $NF)}' file.txt
0 565:10:1 565:12:1 565:23:1 565:18:1 565:17:1 565:25:1
1 564:7:1 564:12:1 564:13:1 564:16:1 564:18:1 564:40:1 564:29:1 564:15:1

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.