# Using uniq -f 1 gives unexpected results

After asking a question on ubuntuforums.org and not getting a satisfying answer, I've decided to ask the question again here on Ask Ubuntu. I need the answer to be very detailed. Specifically, I need to know which lines get compared every time a line is printed using uniq for the following two examples:

file1.txt:

``````\$ cat -A file1.txt
aaa^Iupc\$
b\$
c\$
aaa^Iztp\$
b\$
c\$
C\$
A\$
B\$
B\$
b\$

\$ sort file1.txt | uniq -f 1
A
aaa    upc
aaa    ztp
b
``````

and file2.txt:

``````\$ cat -A file2.txt
aaa^Iupc\$
b\$
c\$
aaa^Iztp\$
b\$
c\$
C\$
A\$
B\$
B\$
bbb^Ixpz\$

\$ sort file2.txt | uniq -f 1
A
aaa    upc
aaa    ztp
b
bbb    xpz
c
``````

I'm confused about the second example. I don't understand how come uppercase B doesn't end up in the final output. Shouldn't the line with uppercase B be printed given that lines `B` and `bbb xpz` are both adjacent to each other? If:

``````B ---> (empty)
``````

and

``````bbb ---> xpz
``````

an empty value and `xpz` are both unique so both lines should be printed. Or am I missing something?

The answer lies in the sorting order and what does `uniq` uses for a field value when less than the given field number (`N`) exists while using `-f N`.

As seen you have ASCII charsets, so the sorting order is much predictable:

``````% sort file.txt
A
aaa upc
aaa ztp
b
b
B
B
bbb xpz
c
c
C
``````

Now, let's use `uniq -f 1` to get unique lines with skipping the (whitespace separated) first field of each line while checking:

``````% sort file.txt | uniq -f 1
A
aaa upc
aaa ztp
b
bbb xpz
c
``````

Now, the important thing to note that, `uniq` uses null string for lines that has has less than the fields mentioned, 1 in this case; so, all the lines that has only one field would be treated as having null strings for other fields while comparing with other lines having >=2 fields.

So, from the `sort file2.txt` output:

``````b
b
B
B
``````

would all be treated as same and only the first line containing `b` would be preserved, hence you have a `b` in the output.

Similarly, from:

``````c
c
C
``````

only the first `c` would end up in the `uniq`'s output.

• OK, but how come typing sort file2.txt | uniq -f 1 yields uppercase A and lowercase b in the second file? Shouldn't both be left out of comparison and not influence the output? Jun 9, 2017 at 1:56
• @John_Patrick_Mason Expanded, check now. Jun 9, 2017 at 2:20
• I think I get it now. I replaced every alphabetical character that has only one field with the letter z to better visualize what's happening. If I understand correctly, the lines that have only one field mess up what uniq sees. If I'm correct, it's as if I hadn't used the sort command at all and passed through uniq an unsorted file. Jun 9, 2017 at 3:19
• @John_Patrick_Mason No, not quite. Without `sort`, you would not get the liens sorted at the first place. SO, only the successive lines that have single field in the original file would result in only the first line. Jun 9, 2017 at 3:22
• @heemaul When I said I replaced every alphabetical character, with only one field, with the letter z, to better visualize, I meant that I created a new file, first, based on the output of: sort file2.txt. Does that make sense? Jun 9, 2017 at 3:46

Here's a table that may help you to work through the process:

``````----------------+---------------+----------+----------------+
sort        |     Remove    | Adjacent |                |
(C locale)     |    field #1   |  match?  |    Output      |
----------------+---------------+----------+----------------+
A               |               |    N*    |A               |
B               |               |    Y     |                |
B               |               |    Y     |                |
C               |               |    Y     |                |
aaa     upc     |   upc         |    N     |aaa     upc     |
aaa     ztp     |   ztp         |    N     |aaa     ztp     |
b               |               |    N     |b               |
b               |               |    Y     |                |
bbb     xpz     |   xpz         |    N     |bbb     xpz     |
c               |               |    N     |c               |
c               |               |    Y     |                |
----------------+---------------+----------+----------------+
* the first line has no adjacent above, so is always output
``````