The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [uniq]

The uniq command deletes repeated lines in a file. The uniq command reads either standard input or a file specified by the InFile parameter

Filter by
Sorted by
Tagged with
1
vote
1answer
28 views

Why does uniq seem to keep some consecutive identical lines?

Without uniq: amin@ubuntu:~/Desktop$ cut -f 1 info.log | tail -n +2 | head -n -1 | sort Abol Abol Ahmad Akbar Arash Hadi Hamed Mahmood Maryam Maryam Mohsen NIma Rasool Sadegh Sepide Sepide With ...
19
votes
8answers
2k views

How to count occurrences of text in a file?

I have a log file sorted by IP addresses, I want to find the number of occurrences of each unique IP address. How can I do this with bash? Possibly listing the number of occurrences next to an ip, ...
10
votes
3answers
2k views

Why did this command put a whitespace at the beginning?

I have this code in a shell script: sort input | uniq -c | sort -nr > output The input file had no preceding white spaces, but the output does. How do I fix this? This is in bash
4
votes
1answer
539 views

How can I find duplicate in the first column, then remove concerning whole lines ?

I have one xlsx file (110725x9 matrix) and I saved as type text (tab delemited) because I don't know whether Unix helps for xlsx files or not. Duplicates rows are always successive line by line. For ...
0
votes
2answers
31 views

Can uniq be used to delete one duplicate line while retaining the other?

I have a file of numbers on different lines. There are many duplications. I would like to delete one line of the duplications while keeping the other line. uniq -d deletes both lines. Is there a way ...
-1
votes
1answer
66 views

uniq treats lines as if they were different but they are ostensibly the same

I created and edited a file with vi called sort.txt which looks like this: sentence one sentence two sentence one sentence three sentence two sentence two sentence three sentence one sentence two ...
1
vote
2answers
89 views

Using uniq -f 1 gives unexpected results

After asking a question on ubuntuforums.org and not getting a satisfying answer, I've decided to ask the question again here on Ask Ubuntu. I need the answer to be very detailed. Specifically, I need ...
4
votes
1answer
360 views

How to get only one item of the a permutation from a list of strings

I want to keep only one permutation among a set of strings and count the occurence of each permutation. To make things more clear, I want to transform (for instance) ab acf ba cfa acf into 2 ab 3 ...
1
vote
1answer
866 views

How does uniq work? [duplicate]

Do not confuse this question to be a duplicate of "what is difference b/w sort -u and sort | uniq" This is in essence a word count program Confusion raised from the following command is reason for ...
0
votes
1answer
73 views

FInd out duplicate lines

I have a file which contain following content. Hi abcd Hi abc hello hello xyz hello I want the find out duplicate lines as well as howmany times it is repeteing.My expected output is as below. 2 Hi ...
-1
votes
1answer
623 views

How long it will take to sort uniq a 62GB file? [closed]

For 5 hours I ran the command sort file1 | uniq > file2 on a 62GB file and I'm just wondering how long it will take. I have a Intel® Core™ i7-4510U CPU @ 2.00GHz × 4 with 8GB RAM. It will be ...
2
votes
1answer
272 views

How to compare two (vague) file lists and print the duplicates?

I have two file lists. backup.txt and backup2.txt Some of the entries aren't exact, so it makes it difficult to find the duplicates with diff or uniq. Example: :::backup.txt::: auser_backup ...
22
votes
3answers
21k views

uniq command not working properly?

So I'm checking the md5 hash of my files with this as my output: 657cf4512a77bf47c39a0482be8e41e0 ./dupes2.txt 657cf4512a77bf47c39a0482be8e41e0 ./dupes.txt 8d60a927ce0f411ec94ac26a4785f749 ./...
-1
votes
3answers
2k views

How to grep keywords with unique value? [closed]

I just want data those have unique value, say we write log file for mobile users <mobile_number1>|20141006 06:15:26||AKQY6LYACZAA4O|12|3|BIHAR| <mobile_number2>|20141006 06:16:05||...
0
votes
2answers
4k views

Sorting two columns with sort and uniq

I have a file that needs sorting in a specific way but I can't figure out how to do it. I've tried the following command but it doesn't give me what I want: sort -t" " -k1,1 Original_File | uniq -...
1
vote
1answer
131 views

uniq --count command is yields incorrect result?

From the document said that the option --count going to prefix lines by the number of occurrences but after I do some test it seems doesn't work. $ echo -e "a\na\b\nc" |uniq --count 1 a ...