Remove all but every 12th file

Question

I have a few thousand files in the format filename.12345.end . I only want to keep every 12th file, so file.00012.end, file.00024.end ... file.99996.end and delete everything else.

The files may also have numbers earlier in their filename, and are normally of the form: file.00064.name.99999.end

I use Bash shell and can't figure out how to loop over the files and then get out the number and check whether it is number%%12=0 deleting the file if not. Can anyone help me?

Thank you, Dorina

Answer 1

Here's a Perl solution. This should be much faster for thousands of files:

perl -e '@bad=grep{/(\d+)\.end/ && $1 % 12 != 0}@ARGV; unlink @bad' *

Which can be further condensed into:

perl -e 'unlink grep{/(\d+)\.end/ && $1 % 12 != 0}@ARGV;' *

If you have too many files and can't use the simple *, you can do something like:

perl -e 'opendir($d,"."); unlink grep{/(\d+)\.end/ && $1 % 12 != 0} readdir($dir)'

As for speed, here's a comparison of this approach and the shell one provided in one of the other answers:

$ touch file.{01..64}.name.{00001..01000}.end
$ ls | wc
  64000   64000 1472000
$ time for f in ./* ; do file="${f%.*}"; if [[ $((10#${file##*.} % 12)) -ne 0 ]]; then rm "$f"; fi; done

real    2m44.258s
user    0m9.183s
sys     1m7.647s

$ touch file.{01..64}.name.{00001..01000}.end
$ time perl -e 'unlink grep{/(\d+)\.end/ && $1 % 12 != 0}@ARGV;' *

real    0m0.610s
user    0m0.317s
sys     0m0.290s

As you can see, the difference is enormous, as expected.

Explanation

• The -e is simply telling perl to run the script given on the command line.

• @ARGV is a special variable containing all the arguments given to the script. Since we're giving it *, it will contain all the files (and directories) in the current directory.

• The grep will search through the list of file names and look for any that match a string of numbers, a dot and end (/(\d+)\.end/).

• Because the numbers (\d) are in a capture group (parentheses), they are saved as $1. So the grep will then check whether that number is a multiple of 12 and, if it isn't, the file name will be returned. In other words, the array @bad holds the list of files to be deleted.

• The list is then passed to unlink() which removes files(but not directories).

Answer 2

Given that your filenames are in the format file.00064.name.99999.end, we first need to trim away everything except our number. We'll use a for loop to do this.

We also need to tell the Bash shell to use base 10, because Bash arithmetic will treat them numbers beginning with a 0 as base 8, which will mess things up for us.

As a script, to be launched when in the directory containing files use:

#!/bin/bash

for f in ./*
do
  if [[ -f "$f" ]]; then
    file="${f%.*}"
    if [[ $((10#${file##*.} % 12)) -ne 0 ]]; then
      rm "$f"
    fi
  else
    echo "$f is not a file, skipping."
  fi
done

Or you can use this very long ugly command to do the same thing:

for f in ./* ; do if [[ -f "$f" ]]; then file="${f%.*}"; if [[ $((10#${file##*.} % 12)) -ne 0 ]]; then rm "$f"; fi; else echo "$f is not a file, skipping."; fi; done

To explain all of the parts:

• for f in ./* means for everything in the current directory, do.... This sets each file or directory found as the variable $f.
• if [[ -f "$f" ]] checks whether the item found is a file, if not we skip to the echo "$f is not... part, which means we don't start deleting directories accidentally.
• file="${f%.*}" sets the $file variable as the filename trimming off whatever comes after the last ..
• if [[ $((10#${file##*.} % 12)) -eq 0 ]] is where the main Arithmetic kicks in. The ${file##*.} trims everything before the last . in our filename without extension. $(( $num % $num2 )) is the syntax for Bash arithmetic to use the modulo operation, the 10# at the start tells Bash to use base 10, to deal with those pesky leading 0s. $((10#${file##*.} % 12)) then leaves us the remainder of our filenames number divided by 12. -ne 0 checks whether the remainder is "not equal" to zero.
• If the remainder is not equal to 0, the file is deleted with the rm command, you may want to replace rm with echo when first running this, to check that you get the expected files to delete.

This solution is non-recursive, meaning that it will only process files in the current directory, it won't go into any sub-directories.

The if statement with the echo command to warn about directories is not really necessary as rm on it's own will complain about directories, and not delete them, so:

#!/bin/bash

for f in ./*
do
  file="${f%.*}"
  if [[ $((10#${file##*.} % 12)) -ne 0 ]]; then
    rm "$f"
  fi
done

Or

for f in ./* ; do file="${f%.*}"; if [[ $((10#${file##*.} % 12)) -ne 0 ]]; then rm "$f"; fi; done

Will work correctly too.

Answer 3

You can use Bash bracket expansion to generate names containing every 12th number. Let's create some test data

$ touch file.{0..9}{0..9}{0..9}{0..9}{0..9}.end # create test data
$ mv file.00024.end file.00024.end.name.99999.end # testing this form of filenames

Then we can use the following

$ ls 'file.'{00012..100..12}* # print these with numbers less than 100
file.00012.end                 file.00036.end  file.00060.end  file.00084.end
file.00024.end.name.99999.end  file.00048.end  file.00072.end  file.00096.end
$ rm 'file.'{00012..100000..12}* # do the job

Works hopelessly slow for large amount of files though - it takes time and memory to generate thousands of names - so it's more a trick that actual efficient solution.

Answer 4

A little bit long, but is what came to my mind.

 for num in $(seq 1 1 11) ; do
     for sequence in $(seq -f %05g $num 12 99999) ; do
         rm file.$sequence.end.99999;
     done
 done

Explanation: Delete every 12th file eleven times.

Answer 5

In all humbleness I think this solution is a lot nicer than the other answer:

find . -name '*.end' -depth 1 | awk 'NR%12 != 0 {print}' | xargs -n100 rm

A little explanation: First we generate a list of files with find. We get all files whose name ends with .end and which are at a depth of 1 (that is to say, they're directly in the working directory and not in any subfolders. You can leave that out if there are no subfolders). The output list will be sorted alphabetically.

Then we pipe that list into awk, where we use the special variable NR which is the line number. We leave out every 12th file by printing the files where NR%12 != 0. The awk command can be shortened to awk 'NR%12', because the result of the modulo operator gets interpreted as a boolean value and the {print} is implicitly done anyway.

So now we have a list of files that need to be deleted, which we can do with xargs and rm. xargs runs the given command (rm) with the standard input as arguments.

If you have many files, you'll get an error saying something like 'argument list too long' (on my machine that limit is 256 kB, and the minimum required by POSIX is 4096 bytes). This can be avoided by the -n 100 flag, which splits the arguments up every 100 words (not lines, something to watch out for if your file names have spaces) and executes a separate rm command, each with only 100 arguments.

Answer 6

For using only bash, my first approach would be to: 1. move all files you want to keep into another directory (ie. all those whose number in filename is a multiple of 12) then 2. delete all remaining files in the directory, then 3. put the multiple-of-12 files you kept back where they were. So something like this might work:

cd dir_containing_files
mkdir keep_these_files
n=0
while [ "${n}" -lt 99999 ]; do
  padded_n="`echo -n "00000${n}" | tail -c 5`"
  mv "filename${padded_n}.end" keep_these_files/
  n=$[n+12]
done
rm filename*.end
mv keep_these_files/* .
rmdir keep_these_files