Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have my references as a text file with a long list of entries and each has two (or more) fields.

The first column is the reference's url; the second column is the title which may vary a bit depending on how the entry was made. Same for third field which may or may not be present.

I want to identify but not remove entries that have the first field (reference url) identical. I know about sort -k1,1 -u but that will automatically (non-interactively) remove all but the first hit. Is there a way to just let me know so I can choose which to retain?

In the extract below of three lines that have the same first field (http://unix.stackexchange.com/questions/49569/), I would like to keep line 2 because it has additional tags (sort, CLI) and delete lines #1 and #3:

http://unix.stackexchange.com/questions/49569/  unique-lines-based-on-the-first-field
http://unix.stackexchange.com/questions/49569/  Unique lines based on the first field   sort, CLI
http://unix.stackexchange.com/questions/49569/  Unique lines based on the first field

Is there a program to help identify such "duplicates"? Then, I can manually clean up by personally deleting lines #1 and #3?

share|improve this question
I don't quite understand your example... could you give a more simplified version of the input and the expected output? –  Oli Mar 15 '14 at 10:58
Please see if it is clearer now? –  DK Bose Mar 15 '14 at 11:09

5 Answers 5

up vote 4 down vote accepted

If I understand your question, I think that you need something like:

for dup in $(sort -k1,1 -u file.txt | cut -d' ' -f1); do grep -n -- "$dup" file.txt; done


for dup in $(cut -d " " -f1 file.txt | uniq -d); do grep -n -- "$dup" file.txt; done

where file.txt is your file containing data about you are interested.

In the output you will see the number of the lines and lines where first field is found two or more times.

share|improve this answer
Thank you: even cut -d " " -f1 file.txt | uniq -d gives me nice output. –  DK Bose Mar 15 '14 at 11:41
@DKBose There are probably more possibilities, but I wanted to use and your command also. –  Radu Rădeanu Mar 15 '14 at 11:44
Thanks. The second command is the one I like. You can remove the first. And if you explain the code that would be nice also :) –  DK Bose Mar 15 '14 at 16:35
Thanks for the edit, good catch :) –  terdon Mar 16 '14 at 4:56
@terdon I thank you :) –  Radu Rădeanu Mar 16 '14 at 5:57

This is a classical problem that can be solved with the uniq command. uniq can detect duplicate consecutive lines and remove duplicates (-u, --unique) or keep duplicates only (-d, --repeated).

Since ordering of duplicate lines is not important for you, you should sort it first. Then use uniq to print unique lines only:

sort yourfile.txt | uniq -u

There is also a -c (--count) option that prints the number of duplicates for the -d option. See the manual page of uniq for details.

If you really do not care about the parts after the first field, you can use the following command to find duplicate keys and print each line number for it (append another | sort -n to have the output sorted by line):

 cut -d ' ' -f1 .bash_history | nl | sort -k2 | uniq -s8 -D

Since you want to see duplicate lines (using the first field as key), you cannot directly use uniq. The issue that make automation difficult is that the title parts vary, but a program cannot automatically determine which title should be considered the final one.

Here is an AWK script (save it to script.awk) that takes your text file as input and prints all duplicate lines so you can decide which to delete. (awk -f script.awk yourfile.txt)

#!/usr/bin/awk -f
    # Store the line ($0) grouped per URL ($1) with line number (NR) as key
    lines[$1][NR] = $0;
    for (url in lines) {
        # find lines that have the URL occur multiple times
        if (length(lines[url]) > 1) {
            for (lineno in lines[url]) {
                # Print duplicate line for decision purposes
                print lines[url][lineno];
                # Alternative: print line number and line
                #print lineno, lines[url][lineno];
share|improve this answer
I think this is close to what I want but I need the opposite of `-f, --skip-fields=N (avoid comparing the first N fields). In other words, I want only the first field, the urls, to be considered. –  DK Bose Mar 15 '14 at 11:20
@DKBose There is a -w (--check-chars) option to limit to a fixed number of characters, but seeing your example, you have variable first fields. Since uniq does not support field selection, you have to use a workaround. I'll include an AWK example since that's easier. –  Lekensteyn Mar 15 '14 at 11:23
Yes, I was just looking at -w but the length of the first field is variable :( –  DK Bose Mar 15 '14 at 11:29
@DKBose Please see the latest edit –  Lekensteyn Mar 15 '14 at 11:39
I'm getting awk: script.awk: line 4: syntax error at or near [ awk: script.awk: line 10: syntax error at or near [ awk: script.awk: line 18: syntax error at or near } –  DK Bose Mar 15 '14 at 11:50

If I read this correctly, all you need is something like

awk '{print $1}' file | sort | uniq -c | 
    while read num dupe; do [[ $num > 1 ]] && grep -n -- "$dupe" file; done

That will print out the number of the line that contains the dupe and the line itself. For example, using this file:

foo bar baz
http://unix.stackexchange.com/questions/49569/  unique-lines-based-on-the-first-field
bar foo baz
http://unix.stackexchange.com/questions/49569/  Unique lines based on the first field   sort, CLI
baz foo bar
http://unix.stackexchange.com/questions/49569/  Unique lines based on the first field

It will produce this output:

2:http://unix.stackexchange.com/questions/49569/  unique-lines-based-on-the-first-field
4:http://unix.stackexchange.com/questions/49569/  Unique lines based on the first field   sort, CLI
6:http://unix.stackexchange.com/questions/49569/  Unique lines based on the first field

To print only the number of the line, you could do

awk '{print $1}' file | sort | uniq -c | 
 while read num dupe; do [[ $num > 1 ]] && grep -n -- "$dupe" file; done | cut -d: -f 1

And to print only the line:

awk '{print $1}' file | sort | uniq -c | 
while read num dupe; do [[ $num > 1 ]] && grep -n -- "$dupe" file; done | cut -d: -f 2-


The awk script just prints the 1st space separated field of the file. Use $N to print the Nth field. sort sorts it and uniq -c counts the occurrences of each line.

This is then passed to the while loop which saves the number of occurrences as $num and the line as $dupe and if $num is greater than one (so it's duplicated at least once) it will search the file for that line, using -n to print the line number. The -- tells grep that what follows is not a command line option, useful for when $dupe can start with -.

share|improve this answer

No doubt the most verbose one in the list, could probably be shorter:

import collections
file = "file.txt"

def find_duplicates(file):
    with open(file, "r") as sourcefile:
        data = sourcefile.readlines()
    splitlines = [
        (index, data[index].split("  ")) for index in range(0, len(data))
    lineheaders = [item[1][0] for item in splitlines]
    dups = [x for x, y in collections.Counter(lineheaders).items() if y > 1]
    dupsdata = []
    for item in dups:
        occurrences = [
            splitlines_item[0] for splitlines_item in splitlines\
                       if splitlines_item[1][0] == item
        corresponding_lines = [
            "["+str(index)+"] "+data[index] for index in occurrences
        dupsdata.append((occurrences, corresponding_lines))

    # printing output   
    print("found duplicates:\n"+"-"*17)
    for index in range(0, len(dups)):
        print(dups[index], dupsdata[index][0])
        lines = [item for item in dupsdata[index][1]]
        for line in lines:
            print(line, end = "")


gives on a textfile like:

monkey  banana
dog  bone
monkey  banana peanut
cat  mice
dog  cowmeat

an output like:

found duplicates:
dog [1, 4]
[1] dog  bone
[4] dog  cowmeat
monkey [0, 2]
[0] monkey  banana
[2] monkey  banana peanut

Once you picked the lines to remove:

removelist = [2,1]

def remove_duplicates(file, removelist):
    removelist = sorted(removelist, reverse=True)
    with open(file, "r") as sourcefile:
        data = sourcefile.readlines()
    for index in removelist:
    with open(file, "wt") as sourcefile:
        for line in data:

remove_duplicates(file, removelist)
share|improve this answer

See the following sorted file.txt:

addons.mozilla.org/en-US/firefox/addon/click-to-play-per-element/ ::: C2P per-element
addons.mozilla.org/en-us/firefox/addon/prospector-oneLiner/ ::: OneLiner
askubuntu.com/q/21033 ::: What is the difference between gksudo and gksu?
askubuntu.com/q/21148 ::: openoffice calc sheet tabs (also askubuntu.com/q/138623)
askubuntu.com/q/50540 ::: What is Ubuntu's Definition of a "Registered Application"?
askubuntu.com/q/53762 ::: How to use lm-sensors?
askubuntu.com/q/53762 ::: how-to-use-to-use-lm-sensors
stackoverflow.com/q/4594319 ::: bash - shell replace cr\lf by comma
stackoverflow.com/q/4594319 ::: shell replace cr\lf by comma
wiki.ubuntu.com/ClipboardPersistence ::: ClipboardPersistence
wiki.ubuntu.com/ClipboardPersistence ::: ClipboardPersistence - Ubuntu Wiki
www.youtube.com/watch?v=1olY5Qzmbk8 ::: Create new mime types in Ubuntu
www.youtube.com/watch?v=2hu9JrdSXB8 ::: Change mouse cursor
www.youtube.com/watch?v=Yxfa2fXJ1Wc ::: Mouse cursor size

Because the list is short, I can see (after sorting) that there are three sets of duplicates.

Then, for example, I can choose to keep:

askubuntu.com/q/53762 ::: How to use lm-sensors?

rather than

askubuntu.com/q/53762 ::: how-to-use-to-use-lm-sensors

But for a longer list this will be difficult. Based on the two answers one suggesting uniq and the other suggesting cut, I find that this command gives me the output I would like:

$ cut -d " " -f1 file.txt | uniq -d
share|improve this answer
I have updated my answer with another variant of cut. If you are doing de-duplicating work, then line numbers may be very helpful. To print all duplicates, use the -D option instead of -d. –  Lekensteyn Mar 15 '14 at 13:42
I think you better use: for dup in $(cut -d " " -f1 file.txt | uniq -d); do grep -n $dup file.txt; done as in my answer. It will give you a better preview about what you are interested. –  Radu Rădeanu Mar 15 '14 at 14:01

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.