19

I need to cleanup a folder periodically. I get a filelist which contains text, which files are allowed. Now I have to delete all files which are not in this file.

Example:

dont-delete.txt:

dontdeletethisfile.txt
reallyimportantfile.txt
neverdeletethis.txt
important.txt

My folder do clean-up contains this as example:

ls /home/me/myfolder2tocleanup/:

dontdeletethisfile.txt
reallyimportantfile.txt
neverdeletethis.txt
important.txt
this-can-be-deleted.txt
also-waste.txt
never-used-it.txt

So this files should be deleted:

this-can-be-deleted.txt
also-waste.txt
never-used-it.txt

I search something to create a delete command with an option to exclude some files provided by file.

4
  • Is this a homework? – mook765 Sep 28 '16 at 15:01
  • I hope you're not his teacher. lol – Gujarat Santana Sep 28 '16 at 15:34
  • 2
    @gujarat We're not free homework service, so the comment is justified. As for the question itself, it may be useful to others, so it's open so far. – Sergiy Kolodyazhnyy Sep 28 '16 at 16:02
  • @Serg I'm totally agree with you – Gujarat Santana Sep 29 '16 at 1:21
13

The rm command is commented out so that you can check and verify that it's working as needed. Then just un-comment that line.

The check directory section will ensure you don't accidentally run the script from the wrong directory and clobber the wrong files.

You can remove the echo deleting line to run silently.

#!/bin/bash

cd /home/me/myfolder2tocleanup/

# Exit if the directory isn't found.
if (($?>0)); then
    echo "Can't find work dir... exiting"
    exit
fi

for i in *; do
    if ! grep -qxFe "$i" filelist.txt; then
        echo "Deleting: $i"
        # the next line is commented out.  Test it.  Then uncomment to removed the files
        # rm "$i"
    fi
done
3
  • I edited your code to avoid useless use of ls and the useless capturing of the output of grep if all you want to know is whether there was a match or not. I also used fixed-string patterns to avoid escaping issues. – David Foerster Oct 13 '16 at 8:51
  • @DavidFoerster Thanks for the contribution. However, when you changed the while loop to a for loop you inadvertently changed the iteration key from i to f. in the declaration, which broke the code. I fixed it. – L. D. James Oct 13 '16 at 13:35
  • Oops, force of habit. I tend to abbreviate shell variable names for file names as f. ;-P (…and +1 for your answer which I forgot earlier.) – David Foerster Oct 13 '16 at 14:13
10

This python script can do this:

#!/usr/bin/env python3
import os
no_remove = set()
with open('./dont-delete.txt') as f:
     for line in f:
         no_remove.add(line.strip())

for f in os.listdir('.'):
    if f not in no_remove:
        print('unlink:' + f ) 
        #os.unlink(f)

Important part is to uncomment the os.unlink() function.

NOTE: add this script and dont-delete.txt to your dont-delete.txt so that they both are on the list, and keep them in the same directory.

3
  • 1
    I changed your code to use a set instead of a list for O(1) instead of O(n) look-up in the second part. – David Foerster Sep 28 '16 at 15:03
  • thanks for your help, i'm normally a windows guy, but python seams too be cool =) – stefan83 Sep 29 '16 at 5:34
  • 1
    @stefan83: Python runs just as well on Windows. – David Foerster Oct 13 '16 at 8:41
7

Here's a one-liner:

comm -2 -3 <(ls) <(sort dont_delete) | tail +2 | xargs -p rm
  1. ls prints all files in the current directory (in sorted order)
  2. sort dont_delete prints all the files we don't want to delete in sorted order
  3. the <() operator turns a string into a file-like object
  4. The comm commands compares two pre-sorted files and prints out lines on which they differ
  5. using the -2 -3 flags causes comm to only print lines contained in the first file but not the second, which will be the list of files that are safe to delete
  6. the tail +2 call is just to remove the heading of the comm output, which contains the name of the input file
  7. Now we get a list of files to delete on standard out. We pipe this output to xargs which will turn the output stream into a list of arguments for rm. The -p option forces xargs to ask for confirmation before executing.
2
  • thx for your help, now I have my solution ! – stefan83 Sep 29 '16 at 5:35
  • @gardenhead, I tired your code but it removes all files in the directory and keep only the first and the last file in the dont-delete list. do you have any idea for this problem? thanks in advance. – Negar Jul 5 '19 at 11:26
2

FWIW it looks like you can do this natively in zsh, using the (+cmd) glob qualifier.

To illustrate, let's start with some files

 % ls
bar  baz  bazfoo  keepfiles.txt  foo  kazoo

and a whitelist file

 % cat keepfiles.txt
foo
kazoo
bar

First, read the whitelist into an array:

 % keepfiles=( "${(f)$(< keepfiles.txt)}" )

or perhaps better

 % zmodload zsh/mapfile
 % keepfiles=( ${(f)mapfile[./keepfiles.txt]} )

(the equivalent of bash's mapfile builtin - or its synonym readarray). Now we can check whether a key (filename) exists in the array using ${keepfiles[(I)filename]} which returns 0 if no match is found:

 % print ${keepfiles[(I)foo]}
1
 % print ${keepfiles[(I)baz]}
0
 %

We can use this to make a function that returns true if there are no matches for $REPLY in the array:

% nokeep() { (( ${keepfiles[(I)$REPLY]} == 0 )); }

Finally, we use this function as a qualifier in our command:

 % ls *(+nokeep)
baz  bazfoo  keepfiles.txt

or, in your case

 % rm -- *(+nokeep)

(You'll likely want to add the name of the whitelist file itself to the whitelist.)

0
2

Unless the output of ls /home/me/myfolder2tocleanup/ exceeds the maximum shell argument limit ARG_MAX which is around 2MB for Ubuntu, I would suggest the following.


A one line command implementation that will do the job, would be as follows:

  1. Copy the dont-delete.txt file to the directory containing the files to be deleted like so:
cp dont-delete.txt /home/me/myfolder2tocleanup/
  1. cd to the directory containing the files to be deleted like so:
cd /home/me/myfolder2tocleanup/
  1. Do a dry-run to test the command and make it print the names of the files that it detects as to be deleted without actually deleting them, like so:
ls -p | grep -v / | sed 's/\<dont-delete.txt\>//g' | sort | comm -3 - <(sort dont-delete.txt) | xargs echo | tr " " "\n"
  1. If you are satisfied with the output, delete the files by running the command like so:
ls -p | grep -v / | sed 's/\<dont-delete.txt\>//g' | sort | comm -3 - <(sort dont-delete.txt) | xargs rm

Explaination:

  • ls -p will list all the files and directories in the current directory and the option -p will add a / to the directory names.
  • grep -v / will exclude directories by removing all items containing a / in their names.
  • sed 's/\<dont-delete.txt\>//g'will exclude the dont-delete.txt file, so it does not get deleted in the process.
  • sort will, just to make sure, sort the remaining output of ls.
  • comm -3 - <(sort dont-delete.txt) will sort the dont-delete.txt file, compare it to the sorted output of ls and exclude filenames that exist in both.
  • xargs rm will remove all the remaining filenames in the already processed output of ls. This means all the items in the current directory will be removed except for directories, files listed in the dont-delete.txt file and the dont-delete.txt file itself

In the dry-run part:

  • xargs echo will print the files that should be removed.
  • tr " " "\n" will translate spaces into new lines for easier readability.

Notice:

In some cases parsing the output of ls might be better avoided.

1

Assuming that your bash shell has the extglob shopt set to on, here is a somewhat more conservative alternative:

rm !($(tr \\n \| < keep.txt))

(...accompanying @gardenhead's otherwise excellent comm suggestion!)

1

I strongly suggest to use rsync solution posted here; else use below solution with an exceptional condition mentioned.

Assuming there is no whitespace (Spaces/Tabs) in your files' listed in a file called excludelist, then you would do:

find /path/to -type f \( ! -name "excludelist" $(printf ' -a ! -name %s\n' $(< excludelist)) \)

Just add -delete to the command above to delete the files which doesn't exist in the excludelist file. If your find doesn't have -delete option you can use rm with -exec as following:

find /path/to -type f \( ! -name "excludelist" $(printf ' -a ! -name %s\n' $(< excludelist)) \) -exec echo rm {} \;

Or using -exec with + terminator instead.

find /path/to -type f \( ! -name "excludelist" $(printf ' -a ! -name %s\n' $(< excludelist)) \) -exec echo rm {} +

echo is just used to dry-run.

-1

My suggestion is:

sed -e 's/^/\.\//' dont-delete.txt > dont-delete-relative-path.txt
find . -type f -print | grep -Fxvf dont-delete-relative-path.txt | xargs -d'\n' rm

Update 2018-08-07

Example:

1: mkdir /tmp/delete-example && cd /tmp/delete-example
2: touch a b c d
3: echo "./a\n./b\n./dont-delete.txt\n" > dont-delete.txt
4: find . -type f -print | grep -Fxvf dont-delete.txt | xargs -d'\n' rm

Note after line 3 you'll have the dont-delete.txt file with contents:

./a
./b
./dont-delete.txt

(the leading ./ is very important)

The files c and d will be deleted.

3
  • I tried this with a text file of the file names separated by a newline. It ended up deleting all the files in the directory. – Jacques MALAPRADE Aug 2 '18 at 10:30
  • I guess your "keep list" was wrong. – nyxz Aug 7 '18 at 13:06
  • I've added example usage. – nyxz Aug 7 '18 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.