11

So dumb impatient me used the following script on my 19.04 server in an attempt to move a bunch of video files into folders with prefixes :

dirs=(A B C D E F G H I J K L M N O P Q R S T U V W X Y Z)
shopt -s nocasematch

for file in *
do
    for dir in "${dirs[@]}"
    do

     if [ -d "$file" ]; then
      echo 'this is a dir, skipping'
      break
     else
      if [[ $file =~ ^[$dir] ]]; then
       echo "----> $file moves into -> $dir <----"
       mv "$file" "$dir"
       break
      fi
     fi
  done
done

No clue where it went wrong, but instead of moving the files to folders it went to a singular output.. so :

----> a1.ts moves into -> A <----
----> a2.ts moves into -> A <----
----> a3.ts moves into -> A <----
----> a4.ts moves into -> A <----
----> a5.ts moves into -> A <----
----> c1.ts moves into -> C <----
----> c2.ts moves into -> C <----
----> c3.ts moves into -> C <----
----> c4.ts moves into -> C <----
----> c5.ts moves into -> C <----

I luckily stopped the process (CTRL+C) as soon as I noticed it wasn't going as intended and didn't go through the whole folder.

So now I've got those files A and C, which are less than a Gb, and by the looks of it are a SINGLE video.

There are 50Gb unaccounted for in the total disk use of the folder itself, but the overall disk space of the computer has remained the same. Making me think the files aren't deleted ?

Any help appreciated, thanks :)

Edit : the files are actually gone, only the last file to be written remains, all it took was some time for the disk use info to update.. moral of the story, run your scripts on mock files before !

  • 3
    And did the directories named A, B and so on existed before running the script? If not you just renamed the files. All files which names started with a or A has been renamed to A, so only the last renamed file survived, the others are overwritten. To call a variable dir does not create a directory! – mook765 Sep 14 at 8:12
  • 2
    thats the way it interpreted it as well. "last renamed file survived" ha. directories didn't exist, i should have added a 'touch' for each before hand. thanks for clarifying – I'm a TI calculator Sep 14 at 8:15
  • 4
    +1 for ".. moral of the story, run your scripts on mock files before !" – sudodus Sep 14 at 8:37
  • 4
    A tip to avoid problems like this: Use mv "$file" "$dir/", with a trailing /; then if $dir doesn't exist, mv will error instead of rename $file to $dir. Also consider mv -i and mv -n. And always do a mkdir -p before moving, for good measure. – marcelm Sep 14 at 19:27
  • 3
    @sudodus Even better moral, "Always backup your data!". – Jon Bentley Sep 14 at 22:24
15

I think this is the problem: You should have created directories A, B, C ... Z. If you did, the mv command should have moved the files to those directories.

But if not, the mv command moves the files to files with those names, A, B, C ... and I think this is what you did.

To make the shellscript safer, you should make it create the directories (if they are not already there) before you start the moving.

dirs=(A B C D E F G H I J K L M N O P Q R S T U V W X Y Z)

for dir in "${dirs[@]}"
do
 mkdir -p $dir
done

If you want things to get even safer, you can also use mv with the -i option

   -i, --interactive
          prompt before overwrite
  • 1
    would adding touch be a good subsitute to mkdir as to avoid conflicts in case of running the script multiple times ? – I'm a TI calculator Sep 14 at 8:17
  • 2
    touch creates a file if the name does not exist. So it will not do what you want in this case. mkdir -p can handle using the script several times. – sudodus Sep 14 at 8:19
  • 6
    Another simple way you can make mv safer is to get into the habit of adding a trailing slash to the target name when the target is a directory i.e. mv "$file" "$dir/" – steeldriver Sep 14 at 11:56
7

@Sudodus already explained what went wrong, but here's a simpler version of your script for next time:

for letter in {a..z}; do 
    dir=${letter^}
    mkdir -p -- "$dir" 
    mv -- "$letter"* "${letter^^}"* "$dir"/
done

Explanation

  • for letter in {a..z}; do: {a..z} expands to all lower case letters between a and z:

    $ echo {a..z}
    a b c d e f g h i j k l m n o p q r s t u v w x y z
    

    So this will iterate over all lower case letters, saving each as $letter.

  • dir=${letter^} : the syntax ${var^^} returns the contents of the variable $var with the first character in upper case (since this only has one character, that's all we need). So, if $letter is a, then ${letter^^} is A, and therefore $dir will be the upper case version of the current $letter.

  • mkdir -p -- "$dir" : create the directory. If it already exists, do nothing (-p). The -- signifies the end of the options, and is useful to protect against names starting with -.
  • mv -- "$letter"* "${letter^}"* "$dir" : move every file (or directory) to the relevant target.

The problem with this, is that it will also move any directories you may have. It won't move the target directories, because either they don't exist yet, or you will try to move them into themselves, but any existing dirs that aren't the target dir will be moved.

If that's a problem, you will have to do something like this:

for file in *; do 
    if [[ ! -d "$file" ]]; then 
        letter="${file:0:1}"
        dir="${letter^}"
        mkdir -p -- "$dir"
        mv -- "$file" "$dir"/
    fi
done
  • What's the difference between ${letter^} and ${letter^^}, and if they're identical, why use ${letter^^} in place of $dir? – Nic Hartley Sep 15 at 20:00
  • 1
    @NicHartley ${var^} capitalizes the first letter only, while ${var^^} capitalizes all letters. It doesn't make any difference here since $letter only has one letter. – terdon Sep 16 at 5:16
  • This is a perfect answer, except you might want to add an extra layer of carefulness by adding a directory slash to $dir in the mvcommand. (In its present form it will fail if a file preexists with a single-letter uppercase name) – Stig Hemmer Sep 16 at 9:53
  • @StigHemmer whoops, yes indeed. Very good point, thanks. Answer edited. – terdon Sep 16 at 11:59
4

Instead of checking every file against a dictionary array which creates alot of iteration you can match files against patterns.

Very basic sort:

#!/bin/bash

videos=./videos
sorted=./sorted

# sort types link,move.
sort_type=link

find "$videos" -maxdepth 1 -type f \
   \( -name '*.avi' -o -name '*.mkv' -o -name '*.mp4' \) -print0 |

while IFS= read -r -d ''; do

    b=$(basename "$REPLY")
    c=${b::1}

    case $c in
        [a-zA-Z]) label=${c^} ;; [0-9]) label="0-9" ;; *) label="_" ;;
    esac

    [[ ! -d "$sorted/$label" ]] && mkdir -p "$sorted/$label"

    if [[ -L $sorted/$label/$b ]] || [[ -e $sorted/$label/$b ]]; then
        echo "File/link: '$b' exists, skipping."
        continue
    fi

    case $sort_type in
        link)
            ln -rfst "$sorted/$label" -- "$REPLY"
            ;;
        move)
               mv -t "$sorted/$label" -- "$REPLY"
            ;;
    esac
done
  • Maybe i did it wrong but that definetly didn't work, lost ten files lol. I'm having troouble understaging the REPLY part? what's the intent All that's left (inside folders A B C D ....) are the aliases files themselves pointing to .. tha alias – I'm a TI calculator Sep 14 at 22:09
  • REPLY set to the line of input read by the read builtin command when no arguments are supplied. – bac0n Sep 14 at 22:36
  • ok and then at the end you do ln -rfst "$sorted/$label" -- "$REPLY" why do the alias if we just moved them with mv ? – I'm a TI calculator Sep 14 at 22:42
  • As default it creates links from "videos" directory into your sorted directory, if you want to mv them you need to uncomment "Move videos" and comment "Link videos" (i think symlinks are less scary) – bac0n Sep 14 at 22:42
  • ...just not both at the same time. – bac0n Sep 14 at 22:52
2

Safeguard in your .bashrc:

alias mv="mv -n --backup=numbered"
  • 1
    @Zanna, Thanks for that! Added quotes. – Arno Nym Ous Sep 15 at 12:47
  • Few questions just to be sure (being a newbie), adding in the .zshrc file is valid too (if using ZSH)? in man mv it says -n Do not overwrite an existing file. (The -n option overrides any previous -f or -i options.) so does it matter that the -n tag will be before following tags? The --backup=numbered will create a double of each right, isn't that a bit overkill (and space energy/consuming) when dealing with uber-large video files (talking terabytes). Thanks ! – I'm a TI calculator Sep 18 at 7:02
2

For the record, some ways to stop mv from overwriting existing files:

  • If you want to move to a directory, append a slash to the target, i.e. use mv "$file" "$dir"/ instead of mv "$file" "$dir". If $dir doesn't exist or isn't a directory, mv will complain:

    $ touch a
    $ mv a z/
    mv: cannot move 'a' to 'z/': Not a directory
    $ touch z
    $ mv a z/
    mv: failed to access 'z/': Not a directory
    

    This seems to make the system call rename("a", "z/"), so it should be safe from time-of-check-to-time-of-use vulnerabilities, in case someone is handling the same set of files at the same time.

  • Alternatively, use mv -t "$dir" "$file". Again, it'll complain if $dir isn't a directory.

  • Use the -n option to prevent overwriting existing files:

    -n, --no-clobber
        do not overwrite an existing file
    

    It won't stop it from renaming the first file, but it won't then trash it with the others.

    This seems to call a plain rename(), so it might be not be safe with simultaneous handling. (There's renameat2() that would support a flag to prevent overwriting.)

1

While apparently not the case for you, it is possible that you could do this and not lose the files. This would require one of two things to be true:

  • One or more 'hard links' to the same files exist elsewhere on the filesystem
  • One or more processes has a file open

Unix filesystems allow more than one directory entry to refer to the exact same file content. This is called a 'hard link'. You can create hard links with the ln command, without the common -s (soft/symbolic) option. As long as at least one hard link exists to the file content, it will not be reused by the filesystem.

(Side note, permissions typically apply to the file content, not to the directory entry. This is why an ordinary user can sometimes delete a file owned by root, but not write to it. The delete operation modifies the folder, not the file itself.)

The filesystem will also not reuse file content as long as at least one process has the file open. Even if no directory entry exists, the filesystem will not consider the space to be free until no processes have it open. The file can be recovered from the virtual filesystem /proc/<pid>/fd by root as long as the file remains open. (Thanks @fluffysheap.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.