how to find number of repeated words in file at starting of the sentence ? e.g.

abc bdbdndnvd hddh hcjdhjc  
dgdgd ghcdggcd abc hjdhcj 
abc ghdsgcgdc cdghcgd dhgch 
hshhj hcdhchd hdjchjd 



only interested in repeated word in whole file at starting only. If that word anywhere else should not be counted.i.e. In above example abc is repeated twice. Can anyone suggest me how can I do this using command ? I am using Ubuntu 16.04.


using cut and uniq

cut -d" " -f1 | sort | uniq -d

The cut command extracts the first word of each line, and sort in combination with uniq -d prints only the duplicated words.


Using awk:

awk '{a[$1]++} END {for (i in a) if (a[i] > 1) print i}'

This just counts the number of times the first word in the line ($1) has been seen and saves it in an array. Then, just loop over all the array elements seen so far and print those which appeared multiple times.

Of course, I have gotten into a rut of looping through arrays in END. steeldriver notes that I don't need to:

awk 'a[$1]++ == 1' file
  • 2
    Based on the OP's problem statement, you could do it on-the-fly by just printing the first occurrence of each repeat awk 'seen[$0]++ == 1' RS='[[:space:]]+' file Mar 14 '17 at 14:08
  • @steeldriver awk 'seen[$1]++ == 1' then?
    – muru
    Mar 14 '17 at 14:09
  • Ooops yes I missed the part about "at starting only" Mar 14 '17 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.