Writing out only files existing in one directory or other or having different size or time of last modify

Question

I am writing a script where I need to compare 2 directories (recursively) and write out only files if they have different size or time of modify(YY-MM-DD HH:MM) or if file exists only in one directory.

Output be in format:

<dir1>:<local-path> <size> <last-modify> <dir2>:<local-path> <size> <last-modify>

If file exists only in one directory:

<dir1>:<local-path> <size> <last-modify>

or

<dir2>:<local-path> <size> <last-modify>

So far I managed to get my data in specified format using:

find dir1 -type f -exec stat -c '%n %s %y' {} \; | sed 's,^[^/]*/,,' | sed 's/\:[^:]*$//' | sort # > dir1.txt
find dir2 -type f -exec stat -c '%n %s %y' {} \; | sed 's,^[^/]*/,,' | sed 's/\:[^:]*$//' | sort # > dir2.txt

Which gives me 2 ordered lists of files in given directories and subdirectories and their size and last modified timestamp.

Now I need to somehow compare them and get them to specified format above. I tried using diff -y but it compares line by line but I need same name to same name. I also tried comm but dont know how to transform that output format.

Any ideas?

Answer 1

I think I'd try to put something together based around using rsync in dry-run mode (--dry-run or -n).

To illustrate, given:

$ tree -Ds Adir/ Bdir/
Adir/
├── [       4096 Nov 19  9:36]  sub1
│   ├── [         35 Nov 19  9:35]  common
│   └── [         23 Nov 19  9:36]  onlyA
├── [       4096 Nov 19  9:41]  sub2
│   ├── [         35 Nov 19  9:35]  common
│   ├── [         44 Nov 19  9:44]  newerA
│   ├── [         44 Nov 19  9:37]  olderA
│   └── [          6 Nov 19 10:36]  size
└── [       4096 Nov 19  9:35]  sub3
    └── [         35 Nov 19  9:35]  common
Bdir/
├── [       4096 Nov 19  9:46]  sub1
│   └── [         35 Nov 19  9:35]  common
├── [       4096 Nov 19 10:36]  sub2
│   ├── [         35 Nov 19  9:35]  common
│   ├── [         44 Nov 19  9:38]  newerA
│   ├── [         44 Nov 19  9:44]  olderA
│   └── [         24 Nov 19 10:36]  size
└── [       4096 Nov 19  9:40]  sub3
    ├── [         35 Nov 19  9:35]  common
    └── [         23 Nov 19  9:40]  onlyB

6 directories, 14 files

then we can list files that have different sizes or modification times as follows:

$ rsync -aOn --delete --itemize-changes Adir/ Bdir/
*deleting   sub3/onlyB
>f+++++++++ sub1/onlyA
>f..t...... sub2/newerA
>f..t...... sub2/olderA
>f.s....... sub2/size

[The change string doesn't really matter for our purposes, but for instance *deleting indicates that sub3/onlyB is not present in the source directory; s indicates a size difference; t indicates a difference in modification time.]

Unfortunately it doesn't seem to be possible to get the actual timestamps directly from the rsync output, but we can simply read the file list and stat the the corresponding files in each directory:

#!/bin/bash

dirA="$1"
dirB="$2"

rsync -aOn --itemize-changes --delete "$dirA"/ "$dirB"/ | while read -r c f ; do
  printf '%s:%s  ' "$dirA" "$(cd "$dirA" && stat -c '%n %s %y' "$f" 2>/dev/null || printf '(none) - - - -')"
  printf '%s:%s\n' "$dirB" "$(cd "$dirB" && stat -c '%n %s %y' "$f" 2>/dev/null || printf '(none) - - - -')"
done

which we can use as follows

$ ./rstat.sh Adir Bdir | column -t
Adir:(none)       -   -           -                   -      Bdir:sub3/onlyB   23  2016-11-19  09:40:12.253318393  -0500
Adir:sub1/onlyA   23  2016-11-19  09:36:52.220421434  -0500  Bdir:(none)       -   -           -                   -
Adir:sub2/newerA  44  2016-11-19  09:44:45.953236221  -0500  Bdir:sub2/newerA  44  2016-11-19  09:38:33.270838033  -0500
Adir:sub2/olderA  44  2016-11-19  09:37:41.675642039  -0500  Bdir:sub2/olderA  44  2016-11-19  09:44:45.953236221  -0500
Adir:sub2/size    6   2016-11-19  10:36:31.460487036  -0500  Bdir:sub2/size    24  2016-11-19  10:36:31.460487036  -0500

Answer 2

I think you already ended up, here is below for:

If file exists only in directory1 (considering any differences in name, size or modification time:

grep -Fxvf dir2.txt dir1.txt > inDir1Only

Or If file exists only in directory2:

grep -Fxvf dir1.txt dir2.txt > inDir2Only

So at the end for your question "write out only files if they have different size or time of modify(YY-MM-DD HH:MM)", just concatenate two above results, like following : )

assuming inDir1Only and inDir2Only content are as following:

$ cat inDir1Only
    c.txt 26 2016-11-04 14:23
    b.txt 26 2016-11-04 14:23

$ cat inDir2Only
    b.txt 57 2016-11-04 18:20
    a.txt 14 2016-11-04 18:11

since of you desired output would be as like below after executing below awk command,

$ awk 'NR==FNR{seen[$1]=$0;next} {
    print "inDir1Only:"$0, ($1 in seen) ?"inDir2Only:"seen[$1]:"";seen[$1]=""}
END{
    for(x in seen) if (seen[x]!=NULL) print "inDir2Only:"seen[x]
}' inDir2Only inDir1Only

inDir1Only:c.txt 26 2016-11-04 14:23 
inDir1Only:b.txt 26 2016-11-04 14:23 inDir2Only:b.txt 57 2016-11-04 18:20
inDir2Only:a.txt 14 2016-11-04 18:11