0

I have a file that looks like that:

2_00003 R034671 31.25   96  55  2   100 195 77  161 7e-07   47.8
2_00003 R034668 31.25   96  55  2   100 195 77  161 7e-07   47.8
2_00003 R034667 31.25   96  55  2   100 195 77  161 7e-07   47.8
2_00003 R034665 31.25   96  55  2   100 195 77  161 7e-07   47.8
2_00003 R034656 34.88   86  45  2   100 185 77  151 9e-07   47.4
2_00003 R034662 34.88   86  45  2   100 185 77  151 9e-07   47.4
2_00003 R034659 34.88   86  45  2   100 185 77  151 9e-07   47.4
2_00004 R014991 31.90   232 141 5   2   232 4   219 5e-28    111
2_00004 R009910 31.90   232 141 5   2   232 4   219 5e-28    111

I want to select just the first lines of the groups that start with the same string in the first column. So for the above file I want to select:

2_00003 R034671 31.25   96  55  2   100 195 77  161 7e-07   47.8
2_00004 R014991 31.90   232 141 5   2   232 4   219 5e-28    111

I thought to use simply

uniq -w 7 <file name>

But I have two problems:

  1. I am not sure if uniq reports always the first occurrence (in my case in the first line)
  2. sometimes in my file the string in the first column has a number of characters higher than 7, so my idea will not work

Any advice?

3

You could consider using sort -u as an alternative to uniq, specifying the first whitespace-separated field as key:

$ sort -uk1,1 file
2_00003 R034671 31.25   96  55  2   100 195 77  161 7e-07   47.8
2_00004 R014991 31.90   232 141 5   2   232 4   219 5e-28    111

Alternatively, you could do something like this in awk:

awk '$1!=last {last=$1; print}' file

which tests the first field of each line ($1) against its last value, and prints the line whenever $1 changes, updating the last value as it goes.

  • Thank you very much!Sorry for the newby question, but using sort can I be sure that it will always take the first line of the input file? Is there a chance that it reorder the first line and it can give me for example the 2 or 3 line of the input file? – efrem Feb 9 '15 at 8:55
  • I did not quite get the awk comand, could you shortly explain it? – efrem Feb 9 '15 at 8:59
  • 2
    I think that sort should leave the order unchanged if the first fields are identical however the awk version is probably safer (and also probably more efficient, since it just makes a single pass - no sorting required). – steeldriver Feb 9 '15 at 13:01
1

Another python approach:

  • read the file
  • list unique occurrences of the first column
  • list the first occurrence in the list
#!/usr/bin/env python3
import sys
file = sys.argv[1]

with open(file) as src:
    lines = src.readlines()
for l in [[l for l in lines if l.startswith(f)][0] for f in set([l.split()[0] for l in lines])]:
    print(l, end = "")

Run it with the textfile as argument:

python3 <script> <text_file>

Note

Although the option above turns out to be a fast one (tested on a file > 1000000 lines), it can be drastically faster (appr. 15% in the tests I ran) if we may assume the string in the first column does not occur on other positions in the records (probably a safe assumption). In that case we can skip the startswith() function:

#!/usr/bin/env python3
import sys
file = sys.argv[1]

with open(file) as src:
    lines = src.readlines()
for l in [[l for l in lines if f in l][0] for f in set([l.split()[0] for l in lines])]:
    print(l, end = "")
  • Your approach is better that my one. +1. – heemayl Feb 7 '15 at 7:54
  • i think you don't need to use sorted function. Op just wants the first line. – Avinash Raj Feb 7 '15 at 7:59
  • @heemayl It is always the challenge here to make it short :) – Jacob Vlijm Feb 7 '15 at 8:10
  • @AvinashRaj absolutely true, removed it. It might be neat to present the first lines sorted though. the uniques were sorted, not the total list of lines. – Jacob Vlijm Feb 7 '15 at 8:11
0

You can do this in a script like this:

first_occurence.sh (set it to be executable)

#!/bin/bash

# Set bash to separate words by newlines only, not spaces
IFS=$'\n'
# read input
input=("$(cat)")

# get a list of unique keys - split input by space with awk for any length
unique_values=($(printf "%s\n" "${input[*]}" | awk -F' ' '{ print $1 }' | uniq))

cur=0

# check each line of input for the key
for line in ${input[@]};
do  
    # wildcard matching
    if [[ "$line" == "${unique_values[$cur]}"* ]]
    then
        # print line if match, and move on to checking the next key
        printf "%s\n" "$line"
        cur=$((cur + 1))
    fi  
    # break the loop if we have used up all of our unique keys (only duplicates remain)
    if [ $cur -ge ${#unique_values[@]} ]
    then
        break
    fi  

done

Run by piping the file in:

./first_occurence.sh < filename 
0

I think steeldriver's solution using sort is the best, although if you want to try something else check the following Python script:

#!/usr/bin/python2
import re
def checking():
        if not check_list:
            result.append(list_of_lines[index - 1])
with open('/path/to/the/file') as f:
    list_of_lines = f.readlines()
    result = []
    for index in range(1, len(list_of_lines)):
        regex_current = re.search('^[0-9]_[0-9]+', list_of_lines[index])
        regex_previous = re.search('^[0-9]_[0-9]+', list_of_lines[index - 1])
        check_list = [x for x in result if x.split()[0] == regex_previous.group()]
        if regex_current == regex_previous:
            checking()
        else:
            checking()
print ''.join(result)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.