2

As per this question and answer: remove file but exclude all files in a list

This code does exactly what I need, apart from it doesn't work on sub directories.

for i in *; do
    if ! grep -qxFe "$i" filelist.txt; then
        echo "Deleting: $i"
        # the next line is commented out.  Test it.  Then uncomment to removed the files
        # rm "$i"
    fi
done

It appears to just match on text in the text file rather than see each new line as a path. So, when the text file contains:

./leaveme.jpg
./i am staying.gif
./james/leaveme.gif
./james/

It still tries to delete the james directory? It also ignores any other files in the james directory which should be deleted.

Is there any way of getting it to recognise full paths in the text file? I have thousands of sub directories so running this script on each individual directory would take forever.

New contributor
ricahrd is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
0

try using rsync with the help of its exclude option as following:

rsync --dry-run -v -r --remove-source-files \
      --exclude-from='/path/to/excludefile.txt' /path/to/source/ /path/to/somewhere/to_delete

notes:

  • the command is running in --dry-run mode; remove it to take real action.
  • before running, make sure your excludefile.txt is located out of /path/to/source path, or add itself to excludefile.txt file to exclude that also from deleting too.
  • rsync cannot remove empty directories if there was any after operation completed, so you will need delete them manually, try

    find /path/to/source -type d -empty -ok rmdir {} \;
    

    change command to below if you don't want confirm for deletion for each directory.

    find /path/to/source -type d -empty -delete
    
  • see here for demo run added

  • and last step, double check the files&directories moved to /path/to/somewhere/to_delete and then delete the entire directory if you were happy with result.

    \rm -rfi /path/to/somewhere/to_delete  # -i switch is used for confirmation, remove it if you don't needed
    
  • Apologies, I did do a demo-run on some temp files last night before I replied but it didn't look like it worked properly - now with a fresh mind today it clearly is working fine, so I obviously can't do late night working like I used to! – ricahrd Jan 15 at 16:15
  • no problem, I'm glad I could help you : ) – αғsнιη Jan 15 at 16:27
0

You can use find . -type f to list all files recursively with their full path. -type f omits all directories and lists only files. But here filelist.txt should contain full paths to the files that we need to exclude from deletion.

IFS=$'\n'
for i in $(find . -type f); do
    if ! grep -qxFe "$i" filelist.txt; then
        echo "Deleting: $i"
        # the next line is commented out.  Test it.  Then uncomment to remove the files
        # rm "$i"
    fi
done
  • Thanks, this works providing I format the paths with ./ at the beginning of each line. However, if there are spaces in a file path it sees each part of that path as a new filename? Is there a way to recognise spaces as part of a path? – ricahrd Jan 14 at 12:25
  • You are right, let me see what I can do – Prathu Baronia Jan 14 at 12:28
  • 1
    @ricahrd Possibly try IFS=$'\n' before starting the for loop. Should solve white space issue in filenames. – Raffa Jan 14 at 14:57
  • @Raffa That seemed to do the trick, thanks. Am wondering if it will fall over when I try it on a location that has 16,000 sub-directories! – ricahrd Jan 14 at 16:00
  • 1
    @PrathuBaronia of course. Either use shopt -s globstar; for f in **; do ... or use find -print0 | while IFS= read -r -d '' f; do ..., or find -exec. The last two are also explained in the link I gave in my previous comment. – terdon Jan 15 at 11:22
0

Asuming your exclude.txt looks like this:

leaveme.jpg
i am staying.gif
james/leaveme.gif

and all sub-directories and files are located in one parent directory and your exclude.txt along with the delete.sh are located outside that parent directory.

The script below will do the following:

  • Read file names from exclude.txt line by line.
  • Find all matching file names under the parent directory and mark them by adding .KEEP suffix to them.
  • Find all files under the parent directory that are not marked with the .KEEP suffix and delete them.
  • Find all empty sub-directories under the parent directory and delete them.
  • Remove the .KEEP suffix from the kept files returning them to their original names.

!!! WARNING: THIS IS NOT A DRY RUN !!!

When run, the script will do all the above actions immediately. Do not run it on your actual directory until you test it first to see if the results are what you want.

To test the script first, run it on a fake copy directory similar in structure and naming to your desired directory.

You can also see what files will be deleted first before actually deleting them by changing this line:

find "$path" -type f ! -name "*.KEEP" -exec rm {} \;

with this line:

find "$path" -type f ! -name "*.KEEP" -exec echo {} "Will be deleted!" \;

in step 2 below to just print the names of the files to be deleted instead of actually deleting them.


To create and use the script, please follow these steps:

  1. Create and edit a script file in your home directory and name it delete.sh by running the following command in the terminal:

    nano ~/delete.sh

  2. Copy and paste the following code into the editor replacing /path/to/parent/directory/ with the full path of the directory that contains the files to be deleted and replacing /path/to/exclude.txt with the full path to your exclude.txt:

#!/bin/bash
path="/path/to/parent/directory/"
filelist="/path/to/exclude.txt"

while IFS=$'\n' read filename
        do
                mv -n "$path$filename" "$path$filename.KEEP"
        done < "$filelist"

find "$path" -type f ! -name "*.KEEP" -exec rm {} \;

find "$path" -type d -empty -delete

while IFS=$'\n' read filename
        do
                mv -n "$path$filename.KEEP" "$path$filename"
        done < "$filelist"


  1. Save the script file and exit the editor by pressing Ctrl + X then press Y.

  2. Make the script file executable by running the following command in the terminal:

    chmod +x ~/delete.sh

  3. Run the script by running the following command in the terminal:

    bash ~/delete.sh

Done, only the files listed in exclude.txt are left under your parent directory.

Your Answer

ricahrd is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.