'Overwrote' files, space still occupied, are they lost?

Question

So dumb impatient me used the following script on my 19.04 server in an attempt to move a bunch of video files into folders with prefixes :

dirs=(A B C D E F G H I J K L M N O P Q R S T U V W X Y Z)
shopt -s nocasematch

for file in *
do
    for dir in "${dirs[@]}"
    do

     if [ -d "$file" ]; then
      echo 'this is a dir, skipping'
      break
     else
      if [[ $file =~ ^[$dir] ]]; then
       echo "----> $file moves into -> $dir <----"
       mv "$file" "$dir"
       break
      fi
     fi
  done
done

No clue where it went wrong, but instead of moving the files to folders it went to a singular output.. so :

----> a1.ts moves into -> A <----
----> a2.ts moves into -> A <----
----> a3.ts moves into -> A <----
----> a4.ts moves into -> A <----
----> a5.ts moves into -> A <----
----> c1.ts moves into -> C <----
----> c2.ts moves into -> C <----
----> c3.ts moves into -> C <----
----> c4.ts moves into -> C <----
----> c5.ts moves into -> C <----

I luckily stopped the process (CTRL+C) as soon as I noticed it wasn't going as intended and didn't go through the whole folder.

So now I've got those files A and C, which are less than a Gb, and by the looks of it are a SINGLE video.

There are 50Gb unaccounted for in the total disk use of the folder itself, but the overall disk space of the computer has remained the same. Making me think the files aren't deleted ?

Any help appreciated, thanks :)

Edit : the files are actually gone, only the last file to be written remains, all it took was some time for the disk use info to update.. moral of the story, run your scripts on mock files before !

Answer 1

I think this is the problem: You should have created directories A, B, C ... Z. If you did, the mv command should have moved the files to those directories.

But if not, the mv command moves the files to files with those names, A, B, C ... and I think this is what you did.

To make the shellscript safer, you should make it create the directories (if they are not already there) before you start the moving.

dirs=(A B C D E F G H I J K L M N O P Q R S T U V W X Y Z)

for dir in "${dirs[@]}"
do
 mkdir -p $dir
done

If you want things to get even safer, you can also use mv with the -i option

   -i, --interactive
          prompt before overwrite

Answer 2

@Sudodus already explained what went wrong, but here's a simpler version of your script for next time:

for letter in {a..z}; do 
    dir=${letter^}
    mkdir -p -- "$dir" 
    mv -- "$letter"* "${letter^^}"* "$dir"/
done

Explanation

• for letter in {a..z}; do: {a..z} expands to all lower case letters between a and z:

  $ echo {a..z}
  a b c d e f g h i j k l m n o p q r s t u v w x y z
  

  So this will iterate over all lower case letters, saving each as $letter.

• dir=${letter^} : the syntax ${var^^} returns the contents of the variable $var with the first character in upper case (since this only has one character, that's all we need). So, if $letter is a, then ${letter^^} is A, and therefore $dir will be the upper case version of the current $letter.

• mkdir -p -- "$dir" : create the directory. If it already exists, do nothing (-p). The -- signifies the end of the options, and is useful to protect against names starting with -.
• mv -- "$letter"* "${letter^}"* "$dir" : move every file (or directory) to the relevant target.

The problem with this, is that it will also move any directories you may have. It won't move the target directories, because either they don't exist yet, or you will try to move them into themselves, but any existing dirs that aren't the target dir will be moved.

If that's a problem, you will have to do something like this:

for file in *; do 
    if [[ ! -d "$file" ]]; then 
        letter="${file:0:1}"
        dir="${letter^}"
        mkdir -p -- "$dir"
        mv -- "$file" "$dir"/
    fi
done

Answer 3

Instead of checking every file against a dictionary array which creates alot of iteration you can match files against patterns.

Very basic sort:

#!/bin/bash

videos=./videos
sorted=./sorted

# sort types link,move.
sort_type=link

find "$videos" -maxdepth 1 -type f \
   \( -name '*.avi' -o -name '*.mkv' -o -name '*.mp4' \) -print0 |

while IFS= read -r -d ''; do

    b=$(basename "$REPLY")
    c=${b::1}

    case $c in
        [a-zA-Z]) label=${c^} ;; [0-9]) label="0-9" ;; *) label="_" ;;
    esac

    [[ ! -d "$sorted/$label" ]] && mkdir -p "$sorted/$label"

    if [[ -L $sorted/$label/$b ]] || [[ -e $sorted/$label/$b ]]; then
        echo "File/link: '$b' exists, skipping."
        continue
    fi

    case $sort_type in
        link)
            ln -rfst "$sorted/$label" -- "$REPLY"
            ;;
        move)
               mv -t "$sorted/$label" -- "$REPLY"
            ;;
    esac
done

Answer 4

Safeguard in your .bashrc:

alias mv="mv -n --backup=numbered"

Answer 5

For the record, some ways to stop mv from overwriting existing files:

• If you want to move to a directory, append a slash to the target, i.e. use mv "$file" "$dir"/ instead of mv "$file" "$dir". If $dir doesn't exist or isn't a directory, mv will complain:

  $ touch a
  $ mv a z/
  mv: cannot move 'a' to 'z/': Not a directory
  $ touch z
  $ mv a z/
  mv: failed to access 'z/': Not a directory
  

  This seems to make the system call rename("a", "z/"), so it should be safe from time-of-check-to-time-of-use vulnerabilities, in case someone is handling the same set of files at the same time.

• Alternatively, use mv -t "$dir" "$file". Again, it'll complain if $dir isn't a directory.

• Use the -n option to prevent overwriting existing files:

  -n, --no-clobber
      do not overwrite an existing file
  

  It won't stop it from renaming the first file, but it won't then trash it with the others.

  This seems to call a plain rename(), so it might be not be safe with simultaneous handling. (There's renameat2() that would support a flag to prevent overwriting.)

Answer 6

While apparently not the case for you, it is possible that you could do this and not lose the files. This would require one of two things to be true:

• One or more 'hard links' to the same files exist elsewhere on the filesystem
• One or more processes has a file open

Unix filesystems allow more than one directory entry to refer to the exact same file content. This is called a 'hard link'. You can create hard links with the ln command, without the common -s (soft/symbolic) option. As long as at least one hard link exists to the file content, it will not be reused by the filesystem.

(Side note, permissions typically apply to the file content, not to the directory entry. This is why an ordinary user can sometimes delete a file owned by root, but not write to it. The delete operation modifies the folder, not the file itself.)

The filesystem will also not reuse file content as long as at least one process has the file open. Even if no directory entry exists, the filesystem will not consider the space to be free until no processes have it open. The file can be recovered from the virtual filesystem /proc/<pid>/fd by root as long as the file remains open. (Thanks @fluffysheap.)