2

For a reason I do not know, Postfix decided to duplicate all the emails I received in a specific folder after my return of vacation. Instead of having ~1000 emails in that folder, I now have 401120! That's way too much for Thunderbird, so the mail client refuse to download any of them.

When I ssh into the mail server, I can see the following:

-rw-------   68 user user     1415 Aug  3 17:45 502716210.M218413P5883.mail.domain.com,S=1415,W=1445:2,
-rw-------   68 user user     1415 Aug  3 17:45 502716327.M661584P5946.mail.domain.com,S=1415,W=1445:2,
-rw-------   68 user user     1415 Aug  3 17:45 502716331.M210364P5948.mail.domain.com,S=1415,W=1445:2,
-rw-------   68 user user     1415 Aug  3 17:45 502716412.M161709P6009.mail.domain.com,S=1415,W=1445:2,
-rw-------   42 user user     2393 Aug  3 17:44 502714577.M151447P4492.mail.domain.com,S=2393,W=2447:2,
-rw-------   42 user user     2393 Aug  3 17:44 502715910.M338737P5564.mail.domain.com,S=2393,W=2447:2,
-rw-------   42 user user     2393 Aug  3 17:44 502715914.M370949P5566.mail.domain.com,S=2393,W=2447:2,
-rw-------   42 user user     2393 Aug  3 17:44 502715964.M213570P5615.mail.domain.com,S=2393,W=2447:2,

I can see that Postfix gave all files their own name, but when I look with nano, I can see that all files with the same size and date are identical.

How can I delete all the duplicate files, to get only one file for each size-date?

1

copy this to a file (say: kill_dupes) and add run rights (chmod a+x kill_dupes) and run it

#!/bin/bash

ls -gGlt | sort -k4,6 -k3,3n > filelist.txt

awk '
BEGIN   {
        printf("#!/bin/bash\n\n");
        filename0 = $7;
        filesize0 = $3;
        filedate0 = $4$5$6;
    }

NF>2    {
        filename1 = $7;
        filesize1 = $3;
        filedate1 = $4$5$6;
        if(filesize1==filesize0 && filedate1==filedate0) {
            printf("rm %s\n", filename1);
        } else {
            filename0 = filename1;
            filesize0 = filesize1;
            filedate0 = filedate1;
        }       
    }
' filelist.txt > rmdupes.sh

then verify the content of rmdupes.sh file

head rmdupes.sh

please check it before actually running it.

(is 502715964.M213570P5615.mail.domain.com,S=2393,W=2447:2 an actual filename?)

  • It worked! Well, mostly. It reduced from 401120 to 46703. But when they are duplicates of different size on the same time, no duplicates are deleted: -rw------- 42 1394 Aug 3 18:13 1502716331.M210367P5948.mail.domain.com,S=1394,W=1424:2, -rw------- 41 2339 Aug 3 18:13 1502716331.M210368P5948.mail.domain.com,S=2339,W=2390:2, -rw------- 42 1394 Aug 3 18:13 1502716412.M161712P6009.mail.domain.com,S=1394,W=1424:2, -rw------- 41 2339 Aug 3 18:13 1502716412.M161713P6009.mail.domain.com,S=2339,W=2390:2, – VincentCrete Aug 14 '17 at 20:42
  • sort added after ls. Please try it this way, it should work now :-) – Macsek Aug 14 '17 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.