remove files and directories but exclude all files from paths in a text file

Question

As per this question and answer: remove file but exclude all files in a list

This code does exactly what I need, apart from it doesn't work on sub directories.

for i in *; do
    if ! grep -qxFe "$i" filelist.txt; then
        echo "Deleting: $i"
        # the next line is commented out.  Test it.  Then uncomment to removed the files
        # rm "$i"
    fi
done

It appears to just match on text in the text file rather than see each new line as a path. So, when the text file contains:

./leaveme.jpg
./i am staying.gif
./james/leaveme.gif
./james/

It still tries to delete the james directory? It also ignores any other files in the james directory which should be deleted.

Is there any way of getting it to recognise full paths in the text file? I have thousands of sub directories so running this script on each individual directory would take forever.

Answer 1

try using rsync with the help of its exclude option as following:

rsync --dry-run -v -r --remove-source-files \
      --exclude-from='/path/to/excludefile.txt' /path/to/source/ /path/to/somewhere/to_delete

notes:

• the command is running in --dry-run mode; remove it to take real action.
• before running, make sure your excludefile.txt is located out of /path/to/source path, or add itself to excludefile.txt file to exclude that also from deleting too.

• rsync cannot remove empty directories if there was any after operation completed, so you will need delete them manually, try

  find /path/to/source -type d -empty -ok rmdir {} \;
  

  change command to below if you don't want confirm for deletion for each directory.

  find /path/to/source -type d -empty -delete
  

• see here for demo run added

• and last step, double check the files&directories moved to /path/to/somewhere/to_delete and then delete the entire directory if you were happy with result.

  \rm -rfi /path/to/somewhere/to_delete  # -i switch is used for confirmation, remove it if you don't needed
  

Answer 2

You can use find . -type f to list all files recursively with their full path. -type f omits all directories and lists only files. But here filelist.txt should contain full paths to the files that we need to exclude from deletion.

IFS=$'\n'
for i in $(find . -type f); do
    if ! grep -qxFe "$i" filelist.txt; then
        echo "Deleting: $i"
        # the next line is commented out.  Test it.  Then uncomment to remove the files
        # rm "$i"
    fi
done

Answer 3

Asuming your exclude.txt looks like this:

leaveme.jpg
i am staying.gif
james/leaveme.gif

and all sub-directories and files are located in one parent directory and your exclude.txt along with the delete.sh are located outside that parent directory.

The script below will do the following:

• Read file names from exclude.txt line by line.
• Find all matching file names under the parent directory and mark them by adding .KEEP suffix to them.
• Find all files under the parent directory that are not marked with the .KEEP suffix and delete them.
• Find all empty sub-directories under the parent directory and delete them.
• Remove the .KEEP suffix from the kept files returning them to their original names.

!!! WARNING: THIS IS NOT A DRY RUN !!!

When run, the script will do all the above actions immediately. Do not run it on your actual directory until you test it first to see if the results are what you want.

To test the script first, run it on a fake copy directory similar in structure and naming to your desired directory.

You can also see what files will be deleted first before actually deleting them by changing this line:

find "$path" -type f ! -name "*.KEEP" -exec rm {} \;

with this line:

find "$path" -type f ! -name "*.KEEP" -exec echo {} "Will be deleted!" \;

in step 2 below to just print the names of the files to be deleted instead of actually deleting them.

---

To create and use the script, please follow these steps:

1. Create and edit a script file in your home directory and name it delete.sh by running the following command in the terminal:

   nano ~/delete.sh

2. Copy and paste the following code into the editor replacing /path/to/parent/directory/ with the full path of the directory that contains the files to be deleted and replacing /path/to/exclude.txt with the full path to your exclude.txt:

#!/bin/bash
path="/path/to/parent/directory/"
filelist="/path/to/exclude.txt"

while IFS=$'\n' read filename
        do
                mv -n "$path$filename" "$path$filename.KEEP"
        done < "$filelist"

find "$path" -type f ! -name "*.KEEP" -exec rm {} \;

find "$path" -type d -empty -delete

while IFS=$'\n' read filename
        do
                mv -n "$path$filename.KEEP" "$path$filename"
        done < "$filelist"

3. Save the script file and exit the editor by pressing Ctrl + X then press Y.

4. Make the script file executable by running the following command in the terminal:

   chmod +x ~/delete.sh

5. Run the script by running the following command in the terminal:

   bash ~/delete.sh

Done, only the files listed in exclude.txt are left under your parent directory.