Ask Ubuntu is a question and answer site for Ubuntu users and developers. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a server which receives a file per client each day into a directory. The filenames are constructed as follows:

uuid_datestring_other-data

For example:

d6f60016-0011-49c4-8fca-e2b3496ad5a7_20160204_023-ERROR
  • uuid is a standard format uuid.
  • datestring is the output from date +%Y%m%d.
  • other-data is variable in length but will never contain an underscore.

I have a file of the format:

#
d6f60016-0011-49c4-8fca-e2b3496ad5a7    client1
d5873483-5b98-4895-ab09-9891d80a13da    client2
be0ed6a6-e73a-4f33-b755-47226ff22401    another_client
...

I need to check that every uuid listed in the file has a corresponding file in the directory, using bash.

I've got this far, but feel like I'm coming from the wrong direction by using an if statement, and that I need to loop through the files in the source directory.

The source_directory and uuid_list variables have been assigned earlier in the script:

# Check the entries in the file list

while read -r uuid name; do
# Ignore comment lines
   [[ $uuid = \#* ]] && continue
   if [[ -f "${source_directory}/${uuid}*" ]]
   then
      echo "File for ${name} has arrived"
   else
      echo "PANIC! - No File for ${name}"
   fi
done < "${uuid_list}"

How should I check that the files in my list exist in the directory? I'd like to use bash functionality as far as possible, but am not against using commands if need be.

share|improve this question
    
Python? And is the server directory "flat"? – Jacob Vlijm Feb 4 at 14:44
    
Yes it's flat, no sub-directories. I'd rather stick with just bash if possible. – Arronical Feb 4 at 14:46
1  
Ok, I won't post. – Jacob Vlijm Feb 4 at 14:46
    
    
I don't really see what's wrong with what you have. You will need to loop through either the UUIDs or the files, why would one loop be better than the other? – terdon Feb 4 at 15:02
up vote 5 down vote accepted

Walk over the files, create an associative array over the uuids contained in their names (I used parameter expansion to extract the uuid). The, read the list, check the associative array for each uuid and report whether the file was recorded or not.

#!/bin/bash
uuid_list=...

declare -A file_for
for file in *_*_* ; do
    uuid=${file%%_*}
    file_for[$uuid]=1
done

while read -r uuid name ; do
    [[ $uuid = \#* ]] && continue
    if [[ ${file_for[$uuid]} ]] ; then
        echo "File for $name has arrived."
    else
        echo "File for $name missing!"
    fi
done < "$uuid_list"
share|improve this answer
1  
Nice (+1), but why is this better than what the OP was doing? You seem to be doing the same basic thing but in two steps instead of one. – terdon Feb 4 at 15:03
1  
@terdon: The main difference is this works :-) The wildcard expansion is done just once, not every time you read a line from the list, which might be faster, too. – choroba Feb 4 at 15:09
    
Yes, that's an important difference. Fair enough :) – terdon Feb 4 at 16:40
    
This is marvellous thank-you, got my +1. Is there any way of including the path to the directory which holds the files? I know I can cd into the directory within the script, but just wondered for the sake of gaining knowledge. – Arronical Feb 4 at 17:26
    
@Arronical: It's possible, but you'll have to remove the path from the string, possible with file=${file##*/}. – choroba Feb 4 at 18:01

Here's a more "bashy" and concise approach:

#!/bin/bash

## Read the UUIDs into the array 'uuids'. Using awk
## lets us both skip comments and only keep the UUID
mapfile -t uuids < <(awk '!/^\s*#/{print $1}' uuids.txt)

## Iterate over each UUID
for uuid in ${uuids[@]}; do
        ## Set the special array $_ (the positional parameters: $1, $2 etc)
        ## to the glob matching the UUID. This will be all file/directory
        ## names that start with this UUID.
        set -- "${source_directory}"/"${uuid}"*
        ## If no files matched the glob, no file named $1 will exist
        [[ -e "$1" ]] && echo "YES : $1" || echo  "PANIC $uuid" 
done

Note that while the above is pretty and will work fine for a few files, its speed depends on the number of UUIDs and will be very slow if you need to process many. If that is the case, either use @choroba's solution or, for something truly fast, avoid the shell and call perl:

#!/bin/bash

source_directory="."
perl -lne 'BEGIN{
            opendir(D,"'"$source_directory"'"); 
            foreach(readdir(D)){ /((.+?)_.*)/; $f{$2}=$1; }
           } 
           s/\s.*//; $f{$_} ? print "YES: $f{$_}" : print "PANIC: $_"' uuids.txt

Just to illustrate the time differences, I tested my bash approach, choroba's and my perl on a file with 20000 UUIDs of which 18001 had a corresponding file name. Note that each test was run by redirecting the script's output to /dev/null.

  1. My bash (~3.5 min)

    real   3m39.775s
    user   1m26.083s
    sys    2m13.400s
    
  2. Choroba's (bash, ~0.7 sec)

    real   0m0.732s
    user   0m0.697s
    sys    0m0.037s
    
  3. My perl (~0.1 sec):

    real   0m0.100s
    user   0m0.093s
    sys    0m0.013s
    
share|improve this answer
    
+1 for a fantastically concise method, this would have to be executed from within the directory containing the files. I know I can cd into the directory in the script, but is there a method whereby the file path could be included in the search? – Arronical Feb 4 at 17:32
    
@Arronical sure, see updated answer. You can use ${source_directory} just like you were doing in your script. – terdon Feb 4 at 17:44
    
Or use "$2" and pass it to the script as a second argument. – alexis Feb 5 at 10:15
    
Check that this runs fast enough for your purposes-- it would be faster to do it with a single directory scan, instead of lots of file lookups like this. – alexis Feb 5 at 10:21
1  
@alexis yes, you're quite right. I did some testing and this becomes very slow if the number of UUIDs/files increases. I added a perl approach (which can be run as a one liner from within th bash script, so technically, still bash if you're open to some creative naming) which is far faster. – terdon Feb 5 at 13:12

This is pure Bash (i.e. no external commands), and it's the most coincise approach that I can think of.

But performance-wise is really not much better than what you currently have.

It will read each line from path/to/file; for each line, it will store the first field in $uuid and prints a message if a file matching the pattern path/to/directory/$uuid* is not found:

#! /bin/bash
[ -z "$2" ] && printf 'Not enough arguments.\n' && exit

while read uuid; do
    [ ! -f "$2/$uuid"* ] && printf '%s missing in %s\n' "$uuid" "$2"
done <"$1"

Call it with path/to/script path/to/file path/to/directory.

Sample output using the sample input file in the question on a test directory hierarchy containing the sample file in the question:

% tree
.
├── path
│   └── to
│       ├── directory
│       │   └── d6f60016-0011-49c4-8fca-e2b3496ad5a7_20160204_023-ERROR
│       └── file
└── script.sh

3 directories, 3 files
% ./script.sh path/to/file path/to/directory
d5873483-5b98-4895-ab09-9891d80a13da* missing in path/to/directory
be0ed6a6-e73a-4f33-b755-47226ff22401* missing in path/to/directory
share|improve this answer
unset IFS
set -f
set +f -- $(<uuid_file)
while  [ "${1+:}" ]
do     : < "$source_directory/$1"*  &&
       printf 'File for %s has arrived.\n' "$2"
       shift 2
done

The idea here is not to worry about reporting errors the shell will report for you. If you try to < open a file which doesn't exist your shell will complain. In fact, it will prepend your script's $0 and the line number on which the error occurred to the error output when it does... This is good information that is provided by default already - so don't bother.

You also don't need to take the file in line-by-line like that - it can be awfully slow. This expands the whole thing in a single shot out to a white-space delimited array of arguments and it handles two at a time. If your data is consistent with your example, then $1 will always be your uuid and $2 will be your $name. If bash can open a match to your uuid - and only one such match exists - then printf happens. Otherwise it doesn't and the shell writes diagnostics to stderr about why.

share|improve this answer
1  
@kos - does the file exist? if not, then it behaves as intended. unset IFS ensures that $(cat <uuid_file) is split on white-space. Shells split on $IFSdifferently when it is comprised of only white-space or is unset. Such split expansions never have any null fields because all white-space sequences stand in as only a single field delimiter. As long as there are only two non-white-space separated fields on each line it should work, i think. in bash, anyway. set -f ensures that the unquoted expansion is not interpreted for globs, and set +f ensures that the later globs are. – mikeserv Feb 5 at 14:46
    
@kos - i just fixed it. I shouldn't have been using <> because that creates a non-existent file. < will report as i meant it to. the possible problem with that though - and the reason i incorrectly used <> in the first place - is that if it is a pipe file without a reader or like a line-buffered char dev it will hang. that could be avoided by handling the error output more explicitly and doing [ -f "$dir/$1"* ]. we are talking about uuids here, and so it should never expand to more than a single file. it is kinda nice though how it reports the failed files names to stderr like that. – mikeserv Feb 5 at 15:14
    
@kos - actually, i suppose i could use ulimit to keep it from creating any files at all and so <> would still be usable that way... <> is better if the glob might expand to a directory because on a linux the read/write will fail and say - thats a directory. – mikeserv Feb 5 at 15:19
    
@kos - oh! I'm sorry - im just being dumb - you have two matches, and so its doing the right thing. i mean for it to error in that way if two matches might be had, these are supposed to be uuids - there should never be a possibility of 2 similar names that match the same glob. thats fully intentional - and it is ambiguous in a way that it shouldnt be. you see what i mean? naming the file for a glob isnt the problem, - special chars arent relevant here - the problem is that bash will only accept a redirection glob if it only matches one file. see man bash under REDIRECTION. – mikeserv Feb 5 at 15:44

The way I'd approach it is to get uuids from file first, then use find

awk '{print $1}' listfile.txt  | while read fileName;do find /etc -name "$fileName*" -printf "%p FOUND\n" 2> /dev/null;done

For readabilty,

awk '{print $1}' listfile.txt  | \
    while read fileName;do \
    find /etc -name "$fileName*" -printf "%p FOUND\n" 2> /dev/null;
    done

Example with a list of files in /etc/, looking for passwd, group,fstab, and THISDOESNTEXIST filenames.

$ awk '{print $1}' listfile.txt  | while read fileName;do find /etc -name "$fileName*" -printf "%p FOUND\n" 2> /dev/null; done
/etc/pam.d/passwd FOUND
/etc/cron.daily/passwd FOUND
/etc/passwd FOUND
/etc/group FOUND
/etc/iproute2/group FOUND
/etc/fstab FOUND

Since you've mentioned the directory is flat,you could use the -printf "%f\n" option to just print filename itself

What this doesn't do is to list missing files. find's small disadvantage is that it doesn't tell you if it doesn't find a file, only when it matches something. What one could do , however , is to check the output - if the output is empty , then we have a file missing

awk '{print $1}' listfile.txt  | while read fileName;do RESULT="$(find /etc -name "$fileName*" -printf "%p\n" 2> /dev/null )"; [ -z "$RESULT"  ] && echo "$fileName not found" || echo "$fileName found"  ;done

More readable:

awk '{print $1}' listfile.txt  | \
   while read fileName;do \
   RESULT="$(find /etc -name "$fileName*" -printf "%p\n" 2> /dev/null )"; \
   [ -z "$RESULT"  ] && echo "$fileName not found" || \
   echo "$fileName found"  
   done

And here's how it performs as a small script:

skolodya@ubuntu:$ ./listfiles.sh                                               
passwd found
group found
fstab found
THISDONTEXIST not found

skolodya@ubuntu:$ cat listfiles.sh                                             
#!/bin/bash
awk '{print $1}' listfile.txt  | \
   while read fileName;do \
   RESULT="$(find /etc -name "$fileName*" -printf "%p\n" 2> /dev/null )"; \
   [ -z "$RESULT"  ] && echo "$fileName not found" || \
   echo "$fileName found"  
   done

One could use stat as alternative, since it's a flat directory, but the code bellow won't work recursively for subdirectories if you ever decide to add those:

$ awk '{print $1}' listfile.txt  | while read fileName;do  stat /etc/"$fileName"* 1> /dev/null ;done        
stat: cannot stat ‘/etc/THISDONTEXIST*’: No such file or directory

If we take the stat idea and run with it, we could use the exit code of stat as indication for whether a file exists or not. Effectivelly, we want to do this:

$ awk '{print $1}' listfile.txt  | while read fileName;do  if stat /etc/"$fileName"* &> /dev/null;then echo "$fileName found"; else echo "$fileName NOT found"; fi ;done

Sample run:

skolodya@ubuntu:$ awk '{print $1}' listfile.txt  | \                                                         
> while read FILE; do                                                                                        
> if stat /etc/"$FILE" &> /dev/null  ;then                                                                   
> echo "$FILE found"                                                                                         
> else echo "$FILE NOT found"                                                                                
> fi                                                                                                         
> done
passwd found
group found
fstab found
THISDONTEXIST NOT found
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.