0

is there anyone out there who can help me sort some files with a bash script or terminal commands (or even a little Java program) that can be used to sort and and move files to new folders? I have been trying for days, but haven't figured it out.

I have a folder called "webpages" with hundreds of html files. Roughly speaking, they are divided in three categories, so I will need to run one script three times with different variables or a script that can do all of the sorting and moving at once.

I want to search for certain strings in the files and then send the files that match to new folders. To simplify, some of the web pages are about politics, some about business and some about computers. So, lets say I want to search all the files in the folder "webpages" for the words "election", "stock market" and "open source" and move the files containing the word "election" to a folder called "politics", the files containing the word "stock market" to a folder called "business" and the files containing the term "open source" to a folder called "computers".

Like I said, I have tried to figure it out, but just got laughed at for my efforts. I am no expert. Thank you!

  • 1
    I think find is the command of your choice, but it has a very long and complex manpage. Try googling about that one. Maybe it can search files by their content, maybe not... – Byte Commander Mar 30 '15 at 17:05
  • Thanks for the help ByteCommander and Jacob! The answer I got from Cos 64 solved the problem for me. :) – David Mar 30 '15 at 18:30
2

Suppose you have the following files in the current directory:

  • a/sm1, with the content "a stock market b"
  • b/sm2, with the content "x stock market y"
  • sm3, that does not contain "stock market"
  • destination, a directory where you want to move files containing "stock market".

Let's find all the files (of type f = file) in the current directory ( . ):

$ find . -type f
./a/sm1
./sm3
./b/sm2

But sm3 does not contain "stock market", we don't want it. In this list of files that we have now, let's search for "stock market", and only display the files that match:

$ find . -type f | xargs grep --files-with-matches "stock market"
./a/sm1
./b/sm2

Now let's get each of the files we got, and move them to the destination directory:

$ for f in $(find . -type f | xargs grep --files-with-matches "stock market"); do mv $f destination/; done

Make sure you have a backup before running this, just in case it doesn't move them how you want.

2

I think some plain bash magic might do the trick:

#!/bin/bash
dir1=""
dir2=""
dir3=""
shopt -s nullglob
for i in *.html)
    do if [ "$(grep 'keyword1' $i)" != "" ]; then
        mv -vf "$i" "$dir1"
    elif [ "$(grep 'keyword2' $i)" != "" ]; then
        mv -vf "$i" "$dir2"
    elif [ "$(grep 'keyword3' $i)" != "" ]; then
        mv -vf "$i" "$dir3"
    else
        echo "$i">>nomatch
    fi
done
cat nomatch
  • 1) Any suggestions on a better implementation? 3) Why would the file nomatch grow? It's used as an else condition @terdon – Wilhelm Erasmus Mar 30 '15 at 18:54
  • Thanks, @terdon For 3) though, wouldn't it only grow if a match can't be found? It would cycle through if, elif, elif and then run the else section?? – Wilhelm Erasmus Mar 30 '15 at 20:48
  • Yes but next time you run the program it will add to the file. Since you're using >> the file will grow each time you run the script. – terdon Mar 30 '15 at 22:28
  • And wow, you did your homework! You even added nullglob, well done and +1 :) – terdon Mar 30 '15 at 22:34
  • One more improvement you could make is to save the files that didn't match in an array and print that array at the end of the script. That way you avoid the use of temp files (you also avoid the danger of having a file called nomatch in the directory you run this from) and don't need to clean up afterwards. – terdon Mar 30 '15 at 22:41
2

Here are a few ways of doing what you want:

1. find

find . -iname '*html' -type f -exec grep -q election "{}" \; -and -exec mv {} politics/ \; 

Explanation

Here, we are using find's -exec option:

-exec command ;
      Execute  command;  true  if 0 status is returned.  All following
      arguments to find are taken to be arguments to the command until
      an  argument  consisting of `;' is encountered.  The string `{}'
      is replaced by the current file name being processed

So, the first -exec searches the file (here, represented by {}) for election and the second one will preform the move. The -and ensures that the second -exec is only run if the first was successful, if the file matched the pattern.

2. find & shell.

This is the same basic approach as the one in Cos64's answer but with a few improvements.

find . -iname '*html' -type f -print0 | 
    while IFS= read -r -d '' file; do 
        grep -q election "$file" && mv "$file" politics/
    done

Explanation

  • The find command will find all files (-type f) whose name ends in .html (or .HTML, the -iname is case insensitive) and print them separated by the NULL character. This is needed because file names in *nix systems can contain any character except / and \0 (NULL). So, you can have files with spaces, newlines and any other strange character. These need to be treated specially.
  • while IFS= read -r -d '' file; do ... done : this iterates over the output of find, saving each file as $file. The IFS= sets the input field separator to nothing, which means we can deal with spaces in file names correctly. The -d '' makes it read \0-separated lines and the -r lets it deal with file names containing \.
    • grep -q election "$file" : search the file for the pattern. The -q suppresses normal output and makes the grep silent.
    • && echo mv "$file" politics/ : the && ensures that this command is only run if the previous one (the grep) was successful.

3. Bash.

This script is very similar to the one in @WilhelmErasmus's very good answer with the difference that i) it can take the set of patterns and replacements from the command line and ii) it also finds files in subdirectories.

#!/usr/bin/env bash

## Exit if no arguments were given
[ -z "$1" ] && echo "At least two arguments are needed." >&2 && exit 1
## Collect the arguments
args=("$@")
## Declare the $dirs associative array
declare -A dirs

## Save the arguments given in the $dirs array.
## $# is the number of arguments given, so this
## will iterate over of them, reading two by two.
for ((i=0;i<$#;i+=2)); 
do
    ## The arguments are pairs of patterns and target directories.
    ## Set the value of this pattern to the value of the next argument,
    ## its target directory. 
    dirs[${args[$i]}]="${args[i+1]}"
done

## Ignore globs that match no files
shopt -s nullglob
## This enables ** to match subdirectories
shopt -s globstar
## Find all .html files
for file in **/*{html,htm,HTM,HTML}
do
    matched=0;
    for pat in "${!dirs[@]}"
    do
        ## Does this file match the pattern?
        ## The `-q` suppresses grep's output.
        grep -q "$pat" "$file" && 
        ## Set matched to 1 if the file matches.
        matched=1 &&
        ## If the grep succeeded, move the file
        ## to the corresponding directory
        mv "$file" "${dirs[$pat]}" && 
        ## If the move succeeded, break the loop
        ## and move to the next pattern.
        break 
    done
    ## Report files that didn't match
    [[ "$matched" -eq 0 ]] && printf "No matches for '%s'\n" "$file" >&2
done

Run the script giving it the names of patterns and their targets. For example, with the ones in your question:

bash move_files.sh "election" "politics" "stock market" "business" "open source" "computers" 
  • Thank you!! I am going to set up some files and try your commands and script as soon as I can. I just have to fix an Ubuntu installation that is having some problems before I will have time to try the script. Thank you for the help! I think I might actually have more than a thousand webpages to sort through! :) – David Apr 2 '15 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.