6

I have about 250,000 files in several folders with sub-folders. Looking for a solution how to find duplicate filenames in all folders and their sub-folders. My OS is Ubuntu 22.04, using bash.

I prefer a bash command/script solution. However, suggestions about tools similar to fdupes -r (but checking if filenames are the same, not their content) are welcome too.

About the files and their names:

  • All files are images and have file extensions.
  • The content of the file is not important and may be different.
  • The extensions of the files are not important and may be different.
  • Letter casing for both file names and their extensions is not consistent.
  • Some of the files have more than one . (period) in their filename. Example: file_Name2.1.png
  • The file extensions are 3 or 4 symbols. Example: .png, .JPG, .jpeg

Structure:

The directory structure is pretty simple: ./[YEAR]/[MONTH]/[IMAGE_NAME].[EXTENSION]. For example:

tree -a
.
├── 2022
│   └── 12
│       ├── file1.png
│       └── File2.png
└── 2023
    ├── 01
    │   ├── file1.jpg
    │   ├── file3.png
    │   └── file4.png
    └── 02
        ├── FILE1.png
        ├── FILE4.PNG
        ├── File5.png
        └── File6.png

Expected result:

  • file1:

    ./2022/12/file1.png
    ./2023/01/file1.jpg
    ./2023/02/FILE1.png
    
  • file4:

    ./2023/01/file4.png
    ./2023/02/FILE4.PNG
    
0

2 Answers 2

6

Assuming your paths:

  • don't have newlines
  • have exactly one period in the filenane

You can use something like:

find . -type f | awk -F / '{
  fname = tolower($NF); # Get the filename in lowercase
  sub(/\.[^.]*$/, "", fname); # Strip extension
  paths[fname] = paths[fname] "\n" $0; # Append full path to existing list of filenames separated by newline
  count[fname]++;
} END {
  for (fname in paths)
    if (count[fname] > 1)
      print fname paths[fname] "\n"
}'
0
1

You asked for a Bash solution, so @muru's answer is excellent, but here is a Python script which may be of interest to you or other users.

#!/usr/bin/env python
import os
from glob import iglob
import sys

# some simple checks
if len(sys.argv) < 2:
    sys.exit("Path is required.")
elif len(sys.argv) > 2:
    sys.exit("Too many arguments.")
else:
    path = sys.argv[1]

# add files to list recursively
files = [i for i in iglob(path + "/**", recursive=True) if os.path.isfile(i)]
# make list with case-insensitive filenames (all lower)
ifnames = [os.path.splitext(os.path.basename(f))[0].lower() for f in files]

# create a dictionary with case-insensitive filenames as keys and their indexes
# as values
d = {}
for i, f in enumerate(ifnames):
    if f in d:
        d[f].append(i)
    else:
        d[f] = [i]

# print the full paths of the duplicate files grouped by case-insensitive
# filenames
for k, v in d.items():
    if len(v) > 1:
        print()
        print(k)
        for i in v:
            print(files[i])

After saving the script with a name of your choice (here I will use script.py), run the following command to give it execution rights (replace the path with the actual one):

chmod u+x /path/to/script.py

The script accepts one argument, which is the path to the directory you want to find duplicates in (target directory), and throws error if more or less arguments are provided.

You can run the script as follows:

/path/to/script.py /path/to/target/directory

For your example tree structure, the script will return:

file1
/path/to/target/directory/2022/12/file1.png
/path/to/target/directory/2023/01/file1.jpg
/path/to/target/directory/2023/02/FILE1.png

file4
/path/to/target/directory/2023/01/file4.png
/path/to/target/directory/2023/02/FILE4.PNG

You can also use relative paths. For example, running:

/path/to/script.py .

will search for duplicates in the current directory, and, provided that it's the target directory, will return:

file1
./2022/12/file1.png
./2023/01/file1.jpg
./2023/02/FILE1.png

file4
./2023/01/file4.png
./2023/02/FILE4.PNG

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .