TL;DR
You can do something like this:
file_expr="*.txt"; sort $file_expr | sed 's/^\s*//; s/\s*$//; /^\s*$/d' | uniq -d | while read dup_line; do grep -Hn "^\s*$dup_line\s*$" $file_expr; done| sort -t: -k3 -k1,2 | awk -F: '{ file=$1; line=$2; $1=$2=""; gsub(/(^[ \t]+)|([ \t]+$)/,"",$0); if (prev != "" && prev != $0) printf ("\n"); printf ("\033[0;33m%s (line %s)\033[0m: %s\n", file, line, $0); prev=$0; }'
Result:
a.txt (line 3): 11
a.txt (line 8): 11
b.txt (line 8): 11
a.txt (line 7): 55
b.txt (line 3): 55
Change the content of the variable file_expr
to change the files affected
Explanation
I used the sed
command to trim all trailing and leading whitespaces and remove empty lines, making the uniq -d
get only the lines which are REALLY duplicate...
Then i loop over the duplicate lines (printed one for every match) and i grep
them into the files using the flags -n
(print file and line) and -H
(always show the filename). The expression ^\s*$dup_line\s*$
into the grep
make it match the whole line (so that for example "qwerty11uiop" doesn't match).
As you can see it works both using file globbing...
file_expr="*.txt"; sort $file_expr | sed 's/^\s*//; s/\s*$//; /^\s*$/d' | uniq -d | while read dup_line; do grep -Hn "^\s*$dup_line\s*$" $file_expr; done
Result:
a.txt:3:11
a.txt:8:11
b.txt:8:11
a.txt:7:55
b.txt:3:55
... and literal file names..
file_expr="a.txt b.txt"; sort $file_expr | sed 's/^\s*//; s/\s*$//; /^\s*$/d' | uniq -d | while read dup_line; do grep -Hn "^\s*$dup_line\s*$" $file_expr; done
Result:
a.txt:3:11
a.txt:8:11
b.txt:8:11
a.txt:7:55
b.txt:3:55
Little Tweaks
I then played around a little to make it visually more comfortable... Like this:
file_expr="a.txt b.txt"; sort $file_expr | sed 's/^\s*//; s/\s*$//; /^\s*$/d' | uniq -d | while read dup_line; do grep -Hn "^\s*$dup_line\s*$" $file_expr; done| sort -t: -k3 -k1,2 | awk -F: '{ file=$1; line=$2; $1=$2=""; gsub(/(^[ \t]+)|([ \t]+$)/,"",$0); if (prev != "" && prev != $0) printf ("\n"); printf ("\033[0;33m%s (line %s)\033[0m: %s\n", file, line, $0); prev=$0; }'
Result:
a.txt (line 3): 11
a.txt (line 8): 11
b.txt (line 8): 11
a.txt (line 7): 55
b.txt (line 3): 55
In this last view everything is more "human" and the duplicates are grouped together first by result and then by file (you can see that the results in a.txt
are all together), so it's easier to understand..
The file name and the line now are yellow (\033[0;33m
) to distinguish from the text in the actual line in case of multiline (excuse the word-pun) duplicates
grep -n
shows line numbers. Also, feeding it touniq
one more time can reduce the amount of searching. So, something likesort *.txt | uniq -D | uniq | while read num; do grep -n $num *.txt; done
might work?awk '{print $1}'
works too.sort *.txt | awk '{print $1}' | uniq -D | uniq | while read num; do grep -n $num *.txt; done