I have this folder that is a whopping 700 GB and some of those files are duplicate. A lot of them have the same name, some duplicate, some not. Basically, I have about 30 folders in side that folder, filled with files and more folders. I want to grab all the files in this 700 GB folder recursively and copy it non-recursively to another directory. Any conflicts will be handled based on size. If two files have the same name and are the same size, the file is skipped, but, if two files have the same name and are different sizes, both files are kept. Any way I can do this?
Complicated question, since dupes have to be renamed, but also renamed dupes have to be checked by size to all its (possibly) previously renamed "fellows".
What it does
As far as I can see (and tested), the script below does that. It renames possible duplicates to
[dupe_1]<file> etc (if they are of different size), and checks after each renaming if the file's size matches either the moved dupe or its renamed versions in the target directory.
#!/usr/bin/env python3 import shutil import os #--- set the directory, the same as the first script dr = "/path/to/recursive/source_directory" dr2 = "/path/to/flat/target_directory" #--- def check_dupe(src, trg): return os.stat(src).st_size == os.stat(trg).st_size for root, dirs, files in os.walk(dr): for file in files: # define src, target src = root+"/"+file; trg = dr2+"/"+file if not os.path.exists(trg): # if target does not exist: move file shutil.copy(src, trg) else: n = 1 # rename the file, check after every renaming if file exists, if size differs while os.path.exists(trg): test = check_dupe(src, trg) if test != True: trg = dr2+"/[dupe_"+str(n)+"]"+file n += 1 else: break if test != True: shutil.copy(src, trg)
How to use
- Copy the script into an empty file, save it as
- In the head section of the script, set the paths to the source- and target directory
Run it by the command:
- Please test on a small sample first
- Keep in mind that this will possibly create a huge "flat" directory, that will not be browsable by nautilus.
cd [folder_name] for file in ./*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/* do if [ -e [destination_folder]/"$file" ]; then if [ `du -s "$file" | cut -d ' ' -f 1` -eq `du -s [destination_folder]/"$file" | cut -d ' ' -f 1` ]; then false else cp "$file" [destination_folder]/"$file"... fi else cp "$file" [destination_folder]/ fi echo "$file copied" done