I have about 167k files in single folder(for now) and renamed by this script in here: Renaming bunch of files, but only part of the title .
How can I find duplicate files by their names (only digits in that specific spot) and delete oldest file:
Aaaaaaa.bbb - 0000125 tag tag_tag 9tag Aaaaaaa.bbb - 0000002 tag 9tag Aaaaaaa.bbb - 0000002 tag tag_tag 9tag

All tools that I used didn't provide such functionality so only script can help.

  • So Aaaaaaa.bbb - 0000002 tag 9tag would be a duplicate of Aaaaaaa.bbb - 0000002 tag tag_tag 9tag because of 0000002, correct? – kos Nov 5 '15 at 22:35
  • Yes, that's correct. – Ceslovas Nov 5 '15 at 22:44
  • So, what defines the name? Will it always be in the format of foo.bar - XXX and the name is foo? Will there always be an extension? Will the space before the - always be the first space in the file name? – terdon Nov 10 '15 at 15:43

Below here's a find, sort and awk one-liner.

Basic idea is to list files, sort them numerically (which works, unless Aaaaaaa.bbb and tags are themselves are numbers), and then let awk store each 3rd field of filenames into prev variable, and compare it with current value of field 3. If they match, print a message.

find . -type f -print | sort --numeric | awk '{if(prev == $3) print $0" is duplicate of "$prevEntry}{ prev=$3; prevEntry=$0}'

Below is a small demo:

    $ seq 6 10 | xargs printf "%07d\n" | xargs -I {} touch "Aaaaaaa.bbb - {} tag 9tag" 

    $ seq 00001 00020 | xargs printf "%07d\n" | xargs -I {} echo "Aaaaaaa.bbb - {} tag tag_tag 9tag"

$ find . -type f -print | sort --numeric | awk '{if(prev == $3) print $0" is duplicate of "$prevEntry}{ prev=$3; prevEntry=$0}'

    ./Aaaaaaa.bbb - 0000006 tag tag_tag 9tag is duplicate of ./Aaaaaaa.bbb - 0000006 tag tag_tag 9tag
    ./Aaaaaaa.bbb - 0000007 tag tag_tag 9tag is duplicate of ./Aaaaaaa.bbb - 0000007 tag tag_tag 9tag
    ./Aaaaaaa.bbb - 0000008 tag tag_tag 9tag is duplicate of ./Aaaaaaa.bbb - 0000008 tag tag_tag 9tag
    ./Aaaaaaa.bbb - 0000009 tag tag_tag 9tag is duplicate of ./Aaaaaaa.bbb - 0000009 tag tag_tag 9tag
    ./Aaaaaaa.bbb - 0000010 tag tag_tag 9tag is duplicate of ./Aaaaaaa.bbb - 0000010 tag tag_tag 9tag
  • It did great job on finding and listing dups, the same result I got from your last one-liner(finding gaps, set from > 15 to < 1). The problem is not finding dups, but eliminating them, deleting old files and leaving new ones by creation (download) date. – Ceslovas Nov 6 '15 at 1:19
  • @Ceslovas Ah, I missed that part when I was reading the question . . .OK. I'll try to see what I could do there and edit my answer once I have a solution – Sergiy Kolodyazhnyy Nov 6 '15 at 1:21
  • @Ceslovas Can you please do me a favor. Select two duplicate files that my command above has outputed , and run ` stat --format=%Y FILENAME` on each one separately, and let me know if the files have same output number – Sergiy Kolodyazhnyy Nov 6 '15 at 1:42
  • It's identical, and I double checked it – Ceslovas Nov 6 '15 at 1:50
  • @Ceslovas how about if we replace %Y with %Z option ? – Sergiy Kolodyazhnyy Nov 6 '15 at 2:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.