I've been using the following command line to do recursive checksumming of directories. It seems to get the job done, but still being a newbie I've been wondering, are there any potential problems with doing it this way? Is it possible for this command to miss files or otherwise mess up?

find ./dir/ -type f -exec sha1sum {} \; > files.sha1
  • You could use + instead of \;. That way multiple files are passed to sha1sum instead of starting a new sha1sum for each file. Otherwise it looks good. – Adaephon Mar 26 '14 at 14:19

There's nothing wrong I can see with your approach. You're excluding directories and find will include hidden files by default. Yeah, it's fine.

But I'll offer you an alternative because that's what I do:

shopt -s globstar dotglob
sha1sum **

globstar enables a recursive match for ** and dotglob enables matching hidden files. Between them they expand to all the filenames and sha1sum can parse them all.

The main problem with this approach is it will pass all the filenames off to sha1sum in one fat pile. While this can be somewhat faster at small loads, it will explode if you have too many filenames. I don't know what the cut-off is.

  • 1
    The cutoff is ARG_MAX, the size of a command's arguments,, you can get the value for your system with getconf ARG_MAX. – terdon Mar 26 '14 at 15:42

Python script with hashlib and os.walk

Aside from using find and globstar, python has modules for hashsum calculation and recursive walk through directory tree. Thus, one can write a simple script just as presented below. In fact, this script is pretty much the same as what I've used for this answer with one minor difference.

This script assumes that you want to recursively walk through the current working directory, so make sure you cd to the desired top directory first.

I would also recommend that you save it in ~/bin directory and run source ~/.bashrc prior to usage, since that way, you can just type the name of the script on command-line.

The script gathers all files, including the hidden-ones ( with the leading dot in the filename ).

Script Source

#!/usr/bin/env python3
import os
import sys
from hashlib import sha1

def get_sha1sum(file_path):
    sha1sum = sha1()
    with open(file_path, 'rb') as fd:
        data_chunk = fd.read(1024)
        while data_chunk:
              data_chunk = fd.read(1024)
    return str(sha1sum.hexdigest())

def find_files(treeroot):
    for dir,subdirs,files in os.walk(treeroot):
         for f in files: 
             full_path = os.path.join(dir,f)
             path_sha1sum = get_sha1sum( full_path  )

def main():

if __name__ == '__main__': main()

Demo Run

bash-4.3$ cd Wallpapers/
bash-4.3$ recursive_checksum.py 
c66af072272d2c516e29832d0a86afa0e8e61d8c ./moon_moon.jpg
37829801c48ea0420414fc78de45adb13e4b117f ./wat.png
6cc3dd2541d00aa5fb8fd6ec703d3c7653ce4708 ./hard_drive_wallpapers/hard-drive.jpg
52fbff84cba6bbbfadc5777c1189ec39aef9176a ./hard_drive_wallpapers/hard-drive5.jpg
5bfe52eb8b31f50dc7bd1b1991dcc1d7260ec65e ./hard_drive_wallpapers/hard-drive4.jpg
f2f85eaa24c8c5b82bbedd55f887ea5fc520ac21 ./hard_drive_wallpapers/hard-drive3.jpg
bcdcf278c176fa93557627a33bedebe4e508e27a ./hard_drive_wallpapers/hard-drive2.jpg

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.