1

After asking a question on ubuntuforums.org and not getting a satisfying answer, I've decided to ask the question again here on Ask Ubuntu. I need the answer to be very detailed. Specifically, I need to know which lines get compared every time a line is printed using uniq for the following two examples:

file1.txt:

$ cat -A file1.txt
aaa^Iupc$
b$
c$
aaa^Iztp$
b$
c$
C$
A$
B$
B$
b$

$ sort file1.txt | uniq -f 1
A
aaa    upc
aaa    ztp
b

and file2.txt:

$ cat -A file2.txt
aaa^Iupc$
b$
c$
aaa^Iztp$
b$
c$
C$
A$
B$
B$
bbb^Ixpz$

$ sort file2.txt | uniq -f 1
A
aaa    upc
aaa    ztp
b
bbb    xpz
c

I'm confused about the second example. I don't understand how come uppercase B doesn't end up in the final output. Shouldn't the line with uppercase B be printed given that lines B and bbb xpz are both adjacent to each other? If:

B ---> (empty)

and

bbb ---> xpz

an empty value and xpz are both unique so both lines should be printed. Or am I missing something?

2

The answer lies in the sorting order and what does uniq uses for a field value when less than the given field number (N) exists while using -f N.

As seen you have ASCII charsets, so the sorting order is much predictable:

% sort file.txt            
A
aaa upc
aaa ztp
b
b
B
B
bbb xpz
c
c
C

Now, let's use uniq -f 1 to get unique lines with skipping the (whitespace separated) first field of each line while checking:

% sort file.txt | uniq -f 1
A
aaa upc
aaa ztp
b
bbb xpz
c

Now, the important thing to note that, uniq uses null string for lines that has has less than the fields mentioned, 1 in this case; so, all the lines that has only one field would be treated as having null strings for other fields while comparing with other lines having >=2 fields.

So, from the sort file2.txt output:

b
b
B
B

would all be treated as same and only the first line containing b would be preserved, hence you have a b in the output.

Similarly, from:

c
c
C

only the first c would end up in the uniq's output.

  • OK, but how come typing sort file2.txt | uniq -f 1 yields uppercase A and lowercase b in the second file? Shouldn't both be left out of comparison and not influence the output? – John_Patrick_Mason Jun 9 '17 at 1:56
  • @John_Patrick_Mason Expanded, check now. – heemayl Jun 9 '17 at 2:20
  • I think I get it now. I replaced every alphabetical character that has only one field with the letter z to better visualize what's happening. If I understand correctly, the lines that have only one field mess up what uniq sees. If I'm correct, it's as if I hadn't used the sort command at all and passed through uniq an unsorted file. – John_Patrick_Mason Jun 9 '17 at 3:19
  • @John_Patrick_Mason No, not quite. Without sort, you would not get the liens sorted at the first place. SO, only the successive lines that have single field in the original file would result in only the first line. – heemayl Jun 9 '17 at 3:22
  • @heemaul When I said I replaced every alphabetical character, with only one field, with the letter z, to better visualize, I meant that I created a new file, first, based on the output of: sort file2.txt. Does that make sense? – John_Patrick_Mason Jun 9 '17 at 3:46
0

Here's a table that may help you to work through the process:

----------------+---------------+----------+----------------+
    sort        |     Remove    | Adjacent |                |
 (C locale)     |    field #1   |  match?  |    Output      |
----------------+---------------+----------+----------------+
A               |               |    N*    |A               |
B               |               |    Y     |                |
B               |               |    Y     |                |
C               |               |    Y     |                |
aaa     upc     |   upc         |    N     |aaa     upc     |
aaa     ztp     |   ztp         |    N     |aaa     ztp     |
b               |               |    N     |b               |
b               |               |    Y     |                |
bbb     xpz     |   xpz         |    N     |bbb     xpz     |
c               |               |    N     |c               |
c               |               |    Y     |                |
----------------+---------------+----------+----------------+
* the first line has no adjacent above, so is always output

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.