0

I have this folder that is a whopping 700 GB and some of those files are duplicate. A lot of them have the same name, some duplicate, some not. Basically, I have about 30 folders in side that folder, filled with files and more folders. I want to grab all the files in this 700 GB folder recursively and copy it non-recursively to another directory. Any conflicts will be handled based on size. If two files have the same name and are the same size, the file is skipped, but, if two files have the same name and are different sizes, both files are kept. Any way I can do this?

  • 1
    "if two files have the same name and are different sizes, both files are kept." Tricky. Say you had files bar/a, foo/a, baz/a, bar/a having a different size. Once you copy bar/a to target/a, what do you do to the others? Keep foo/a and discard the other? Keep foo/a as foo/a.1 and compare baz/a to both? – muru Jun 15 '15 at 19:51
  • 1
    how do filenames look like ? I mean , file is original and file[1] is the duplicate ? or is it exactly same name ? Please add more info – Sergiy Kolodyazhnyy Jun 15 '15 at 20:00
  • muru the others are kept. – user245115 Jun 15 '15 at 20:01
  • Serg it's both. There's more then one duplicate for some particular, some having the same name while others are like file(2), file(3), etc. – user245115 Jun 15 '15 at 20:04
  • 2
    I think you should use some duplicate finder first and get rid of them. See fdupes, for one. – muru Jun 15 '15 at 20:14
1

Complicated question, since dupes have to be renamed, but also renamed dupes have to be checked by size to all its (possibly) previously renamed "fellows".

What it does

As far as I can see (and tested), the script below does that. It renames possible duplicates to [dupe_1]<file>, [dupe_1]<file> etc (if they are of different size), and checks after each renaming if the file's size matches either the moved dupe or its renamed versions in the target directory.

#!/usr/bin/env python3
import shutil
import os

#--- set the directory, the same as the first script
dr = "/path/to/recursive/source_directory"
dr2 = "/path/to/flat/target_directory"
#---

def check_dupe(src, trg):
    return os.stat(src).st_size == os.stat(trg).st_size

for root, dirs, files in os.walk(dr):
    for file in files:
        # define src, target
        src = root+"/"+file; trg = dr2+"/"+file
        if not os.path.exists(trg):
            # if target does not exist: move file
            shutil.copy(src, trg)
        else:
            n = 1
            # rename the file, check after every renaming if file exists, if size differs
            while os.path.exists(trg):
                test = check_dupe(src, trg)
                if test != True:
                    trg = dr2+"/[dupe_"+str(n)+"]"+file
                    n += 1
                else:
                    break
            if test != True:
                shutil.copy(src, trg)

How to use

  1. Copy the script into an empty file, save it as reorganize.py
  2. In the head section of the script, set the paths to the source- and target directory
  3. Run it by the command:

    python3 /path/to/reorganize.py
    

Notes

  • Please test on a small sample first
  • Keep in mind that this will possibly create a huge "flat" directory, that will not be browsable by nautilus.
  • I decided to test this by making it copy the source folder to another external drive. Seems to work, some dupes though using different names. Most of the dupes use the same name, rarely any using different names, so I guess it will still free up a lot of space, thanks. – user245115 Jun 15 '15 at 21:13
1

try this:

cd [folder_name]
for file in ./*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*
do
    if [ -e [destination_folder]/"$file" ]; then
        if [ `du -s "$file" | cut -d ' ' -f 1` -eq `du -s [destination_folder]/"$file" | cut -d ' ' -f 1` ]; then
            false
        else
            cp "$file" [destination_folder]/"$file"...
        fi
    else
        cp "$file" [destination_folder]/
    fi
    echo "$file copied"
done
  • Bash has recursive globbing. – muru Jun 15 '15 at 20:31
  • edited. see above. – Daniel Jun 15 '15 at 20:31
  • Daniel, this seems to merely copy all the files. OP wants to copy only unique files based on both names and size. It's not that simple – Sergiy Kolodyazhnyy Jun 15 '15 at 20:32
  • "I want to grab all the files in this 700 GB folder recursively and copy it non-recursively to another directory" That sounds like more than just duplicates... – Daniel Jun 15 '15 at 20:35
  • Yes, and there's more to that. Please re-read the question and the comments bellow the question – Sergiy Kolodyazhnyy Jun 15 '15 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.