Character encoding problem with filenames - find broken filenames

Question

I have the problem described in this Q&A. Probably from quite old linux distros or from windows I have several files with broken filenames. ls displays a "?" instead of the broken character. I successfully renamed some of these files, but I don't know if I've found all of them.

Is there any method to find all affected files?

Answer 1

Assuming you are using utf-8 encoding (the default in Ubuntu), this script should hopefully identify the filenames and rename them for you.

It works by using find with C-encoding (ASCII) to locate files with unprintable characters in them. It then tries to determine if these unprintable characters are utf-8 characters or not. If not, it shows you the filenames decoded with each of the encodings listed in the enc array, allowing you to select the one that looks right in order to rename it.

latin1 was commonly used on older Linux systems, and windows-1252 is commonly used by windows nowadays (I think). iconv -l will show you a list of possible encodings.

#!/bin/bash

# List of encodings to try. (max 10)
enc=( latin1 windows-1252 )

while IFS= read -rd '' file <&3; do
    base=${file##*/} dir=${file%/*}

    # if converting from utf8 to utf8 succeeds, we'll assume the filename is ok.
    iconv -f utf8 <<< "$base" >/dev/null 2>&1 && continue

    # display the filename converted from each enc to utf8
    printf 'In %s:\n' "$dir/"
    for i in "${!enc[@]}"; do
        name=$(iconv -f "${enc[i]}" <<< "$base")
        printf '%2d - %-12s: %s\n' "$i" "${enc[i]}" "$name"
    done
    printf ' s - Skip\n'

    while true; do
        read -p "? " -n1 ans
        printf '\n'
        if [[ $ans = [0-9] && ${enc[ans]} ]]; then
            name=$(iconv -f "${enc[ans]}" <<< "$base")
            mv -iv "$file" "$dir/$name"
            break
        elif [[ $ans = [Ss] ]]; then
            break
        fi
    done
done 3< <(LC_ALL=C find . -depth -name "*[![:print:][:space:]]*" -print0)

Answer 2

Try this:

find / | grep -P "[\x80-\xFF]"

This will locate all non-ASCII characters in file and folder names, and help you to find the guilty culprits :P

Answer 3

Begin with this regex find command and modify it until you hit only those you are interested in: find . | egrep [^a-zA-Z0-9_./-\s].
The one above will find filenames that has a non UTF-8 character(s).

Answer 4

Here's another version, extremely simple, that uses unidecode utility (install: sudo apt install python3-unidecode), which attempts as best it can to convert Unicode characters to "equivalent" ASCII character. Obviously this comes with caveats; see https://pypi.org/project/Unidecode/

It first renames directories, then files (changing directory names and files at the same time might have unpredictable results, doing depth-first is necessary in the case of renaming directories.)

(With a few minor changes you can make it just print out what would be changed, instead of doing the actual changes.)

#!/bin/bash

[ ! -d "$1" ] && echo "** error: expecting directory to search" && exit 2
DIR=$1

do_rename() {
  orig=$1
  new=$(unidecode -c "$1")

  printf "mv -i \"%s\" \"%s\"\n" "$orig" "$new" ## DEBUG
  mv -i "$orig" "$new"
}
export -f do_rename

# directories
LC_ALL=C find $DIR -depth -type d -name '*[! -~]*' -exec bash -c 'do_rename "$0"' {} \;

# files
LC_ALL=C find $DIR -depth -type f -name '*[! -~]*' -exec bash -c 'do_rename "$0"' {} \;