2

As per this question and answer: remove file but exclude all files in a list

This code does exactly what I need, apart from it doesn't work on sub directories.

for i in *; do
    if ! grep -qxFe "$i" filelist.txt; then
        echo "Deleting: $i"
        # the next line is commented out.  Test it.  Then uncomment to removed the files
        # rm "$i"
    fi
done

It appears to just match on text in the text file rather than see each new line as a path. So, when the text file contains:

./leaveme.jpg
./i am staying.gif
./james/leaveme.gif
./james/

It still tries to delete the james directory? It also ignores any other files in the james directory which should be deleted.

Is there any way of getting it to recognise full paths in the text file? I have thousands of sub directories so running this script on each individual directory would take forever.

2
0

try using rsync with the help of its exclude option as following:

rsync --dry-run -v -r --remove-source-files \
      --exclude-from='/path/to/excludefile.txt' /path/to/source/ /path/to/somewhere/to_delete

notes:

  • the command is running in --dry-run mode; remove it to take real action.
  • before running, make sure your excludefile.txt is located out of /path/to/source path, or add itself to excludefile.txt file to exclude that also from deleting too.
  • rsync cannot remove empty directories if there was any after operation completed, so you will need delete them manually, try

    find /path/to/source -type d -empty -ok rmdir {} \;
    

    change command to below if you don't want confirm for deletion for each directory.

    find /path/to/source -type d -empty -delete
    
  • see here for demo run added

  • and last step, double check the files&directories moved to /path/to/somewhere/to_delete and then delete the entire directory if you were happy with result.

    \rm -rfi /path/to/somewhere/to_delete  # -i switch is used for confirmation, remove it if you don't needed
    
5
  • Apologies, I did do a demo-run on some temp files last night before I replied but it didn't look like it worked properly - now with a fresh mind today it clearly is working fine, so I obviously can't do late night working like I used to! – ricahrd Jan 15 '20 at 16:15
  • no problem, I'm glad I could help you : ) – αғsнιη Jan 15 '20 at 16:27
  • this worked brilliantly for a smallish 100GB directory with only 400,000 moves to process, completed in 5 hours. Now I'm trying to do the same with a 1TB directory with 7 million file moves (keeping 1 million), a dry run took 5 days which means I can't complete it over a weekend so will impact work hours. Am trying to figure out if I can also exclude any files modified after a certain date...the rsync manual doesn't seem to show any options for it and most people have to include a find command to do this? Do you think it will be possible to add this to the existing command? – ricahrd Jan 24 '20 at 12:46
  • it depends where are source and destination paths; yes, you can exclude certain files same as you have some before, just use find command to report certain files and append them to your excludefile.txt, also use some options for rsync for better performance like -t, --delete-during, compression (if low bandwidth and slow connection on remote); amother option is if purpose of this task is just delete all files directories except excluded ones, then why not you don't do opposite of activity, saying that move excludefile.txt files or directories and delete all remained files/directories. – αғsнιη Jan 24 '20 at 16:39
  • ... there will many options but I have no such environments to test in real situations. you will also find many suggestions over the Internet for rsync optimization. – αғsнιη Jan 24 '20 at 16:39
0

You can use find . -type f to list all files recursively with their full path. -type f omits all directories and lists only files. But here filelist.txt should contain full paths to the files that we need to exclude from deletion.

IFS=$'\n'
for i in $(find . -type f); do
    if ! grep -qxFe "$i" filelist.txt; then
        echo "Deleting: $i"
        # the next line is commented out.  Test it.  Then uncomment to remove the files
        # rm "$i"
    fi
done
11
  • Thanks, this works providing I format the paths with ./ at the beginning of each line. However, if there are spaces in a file path it sees each part of that path as a new filename? Is there a way to recognise spaces as part of a path? – ricahrd Jan 14 '20 at 12:25
  • You are right, let me see what I can do – Prathu Baronia Jan 14 '20 at 12:28
  • 1
    @ricahrd Possibly try IFS=$'\n' before starting the for loop. Should solve white space issue in filenames. – Raffa Jan 14 '20 at 14:57
  • @Raffa That seemed to do the trick, thanks. Am wondering if it will fall over when I try it on a location that has 16,000 sub-directories! – ricahrd Jan 14 '20 at 16:00
  • 1
    @PrathuBaronia of course. Either use shopt -s globstar; for f in **; do ... or use find -print0 | while IFS= read -r -d '' f; do ..., or find -exec. The last two are also explained in the link I gave in my previous comment. – terdon Jan 15 '20 at 11:22
0

Asuming your exclude.txt looks like this:

leaveme.jpg
i am staying.gif
james/leaveme.gif

and all sub-directories and files are located in one parent directory and your exclude.txt along with the delete.sh are located outside that parent directory.

The script below will do the following:

  • Read file names from exclude.txt line by line.
  • Find all matching file names under the parent directory and mark them by adding .KEEP suffix to them.
  • Find all files under the parent directory that are not marked with the .KEEP suffix and delete them.
  • Find all empty sub-directories under the parent directory and delete them.
  • Remove the .KEEP suffix from the kept files returning them to their original names.

!!! WARNING: THIS IS NOT A DRY RUN !!!

When run, the script will do all the above actions immediately. Do not run it on your actual directory until you test it first to see if the results are what you want.

To test the script first, run it on a fake copy directory similar in structure and naming to your desired directory.

You can also see what files will be deleted first before actually deleting them by changing this line:

find "$path" -type f ! -name "*.KEEP" -exec rm {} \;

with this line:

find "$path" -type f ! -name "*.KEEP" -exec echo {} "Will be deleted!" \;

in step 2 below to just print the names of the files to be deleted instead of actually deleting them.


To create and use the script, please follow these steps:

  1. Create and edit a script file in your home directory and name it delete.sh by running the following command in the terminal:

    nano ~/delete.sh

  2. Copy and paste the following code into the editor replacing /path/to/parent/directory/ with the full path of the directory that contains the files to be deleted and replacing /path/to/exclude.txt with the full path to your exclude.txt:

#!/bin/bash
path="/path/to/parent/directory/"
filelist="/path/to/exclude.txt"

while IFS=$'\n' read filename
        do
                mv -n "$path$filename" "$path$filename.KEEP"
        done < "$filelist"

find "$path" -type f ! -name "*.KEEP" -exec rm {} \;

find "$path" -type d -empty -delete

while IFS=$'\n' read filename
        do
                mv -n "$path$filename.KEEP" "$path$filename"
        done < "$filelist"


  1. Save the script file and exit the editor by pressing Ctrl + X then press Y.

  2. Make the script file executable by running the following command in the terminal:

    chmod +x ~/delete.sh

  3. Run the script by running the following command in the terminal:

    bash ~/delete.sh

Done, only the files listed in exclude.txt are left under your parent directory.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.