71

I would like to create a md5 checksum list for all files in a directory.

I want to cat filename | md5sum > ouptput.txt. I want to do this in 1 step for all files in my directory.

Any assistance would be great.

  • How can I find the md5sum (checksum) of a specific backup file? -What should my script look like -what is the command for it – user309300 Jul 26 '14 at 7:09
103

You can pass md5sum multiple filenames or bash expansions:

$ md5sum * > checklist.chk  # generates a list of checksums for any file that matches *
$ md5sum -c checklist.chk   # runs through the list to check them
cron: OK
database.sqlite3: OK
fabfile.py: OK
fabfile.pyc: OK
manage.py: OK
nginx.conf: OK
uwsgi.ini: OK

If you want to get fancy you can use things like find to drill down and filter the files, as well as working recursively:

find -type f -exec md5sum "{}" + > checklist.chk
  • 1
    ♦ how to use the above for getting md5sum of the files inside sub directories, the above md5sum * emitting going into subdirectory level by saying ... is a directory – Kasun Siyambalapitiya Aug 31 '17 at 6:36
  • Sorry for asking by running find -type f -exec md5sum '{}' + and `find -type f -exec md5sum '{}' ` I was able to get it. Thanks :) – Kasun Siyambalapitiya Aug 31 '17 at 6:40
  • I prefer the look of this output more: openssl md5 * > checklist.txt – ashley Aug 31 '17 at 14:23
11

A great checksum creation/verification program is rhash. It creates even SFV compatible files, and checks them too.

It supports md4, md5, sha1, sha512, crc32 and many many other.

Moreover it can do recursive creation (-r option) like md5deep or sha1deep.

Last but not least you can format the output of the checksum file; for example:

rhash --md5 -p '%h,%p\n' -r /home/

outputs a CSV file including the full path of files recursively starting with the /home directory.

I find extremely useful even the -e option rename files by inserting crc32 sum into name.

You can change "md5sum" with "rhash" in the PhoenixNL72 examples.

  • 1
    What exactly does '/home/' do here? This tool looks pretty good. – Andy Ibanez Nov 11 '17 at 0:49
  • I think it's an error, it certainly errors for me. The -p is the format for the output. I'll correct it. – pbhj Jul 19 '18 at 18:48
7

Here are two more extensive examples:

  1. Create an md5 file in each directory which doesn't already have one, with absolute paths:

    find "$PWD" -type d | sort | while read dir; do [ ! -f "${dir}"/@md5Sum.md5 ] && echo "Processing " "${dir}" || echo "Skipped " "${dir}" " @md5Sum.md5 already present" ; [ ! -f "${dir}"/@md5Sum.md5 ] &&  md5sum "${dir}"/* > "${dir}"/@md5Sum.md5 ; chmod a=r "${dir}"/@md5Sum.md5;done 
    
  2. Create an md5 file in each folder which doesn't already have one: no paths, only filenames:

    find "$PWD" -type d | sort | while read dir; do cd "${dir}"; [ ! -f @md5Sum.md5 ] && echo "Processing " "${dir}" || echo "Skipped " "${dir}" " @md5Sum.md5 allready present" ; [ ! -f @md5Sum.md5 ] &&  md5sum * > @md5Sum.md5 ; chmod a=r "${dir}"/@md5Sum.md5 ;done 
    

What differs between 1 and 2 is the way the files are presented in the resulting md5 file.

The commands do the following:

  1. Build a list of directory names for the current folder. (Tree)
  2. Sort the folder list.
  3. Check in each directory if the file @md5sum.md5 exists. Output Skipped if it exists, output Processing if it doesn't exist.
  4. If the @md5Sum.md5 file doesn't exist, md5Sum will generate one with the checksums of all the files in the folder. 5) Set the generated @md5Sum.md5 file to read only.

The output of this entire script can be redirected to a file (.....;done > test.log) or piped to another program (like grep). The output will only tell you which directories where skipped and which have been processed.

After a successful run, you will end up with an @md5Sum.md5 file in each subdirectory of your current directory

I named the file @md5Sum.md5 so it'll get listed at the top of the directory in a samba share.

Verifying all @md5Sum.md5 files can be done by the next commands:

find "$PWD" -name @md5Sum.md5 | sort | while read file; do cd "${file%/*}"; md5sum -c @md5Sum.md5; done > checklog.txt

Afterwards you can grep the checklog.txt using grep -v OK to get a list of all files that differ.

To regenerate an @md5Sum.md5 in a specific directory, when you changed or added files for instance, either delete the @md5Sum.md5 file or rename it and run the generate command again.

  • Command #1 using full paths: md5sum "${dir}"/* is not picking up hidden files starting with dot. Any remedy ? – user14654 Sep 4 '14 at 13:07
  • Late to the party, but since md5sum wants file names (seems uhappy if you give it a directory) it might be easiest to ask find for help: $ md5sum $( find "$dir" -maxdepth 0 -type f ) The "-type f" pulls regular and .hidden files (also, find has many options, but can be super helpful). Omit the "-maxdepth 0" part if you want everything recursively. Note: this is fragile if the filenames have embedded spaces, then you can read up on xargs and -0 ( en.wikipedia.org/wiki/Xargs ) – jgreve Mar 17 '18 at 3:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.