92

I would like to create a md5 checksum list for all files in a directory.

I want to cat filename | md5sum > ouptput.txt. I want to do this in 1 step for all files in my directory.

Any assistance would be great.

1
  • ``` find . -type f -exec md5sum {} > md5sums.txt \; ``` this is all you need! – george Jan 29 at 20:19
139

You can pass md5sum multiple filenames or bash expansions:

$ md5sum * > checklist.chk  # generates a list of checksums for any file that matches *
$ md5sum -c checklist.chk   # runs through the list to check them
cron: OK
database.sqlite3: OK
fabfile.py: OK
fabfile.pyc: OK
manage.py: OK
nginx.conf: OK
uwsgi.ini: OK

If you want to get fancy you can use things like find to drill down and filter the files, as well as working recursively:

find -type f -exec md5sum "{}" + > checklist.chk
5
  • 1
    ♦ how to use the above for getting md5sum of the files inside sub directories, the above md5sum * emitting going into subdirectory level by saying ... is a directory – Kasun Siyambalapitiya Aug 31 '17 at 6:36
  • Sorry for asking by running find -type f -exec md5sum '{}' + and `find -type f -exec md5sum '{}' ` I was able to get it. Thanks :) – Kasun Siyambalapitiya Aug 31 '17 at 6:40
  • I prefer the look of this output more: openssl md5 * > checklist.txt – ashley Aug 31 '17 at 14:23
  • 1
    If you're using a shell that's happy to evaluate ** recursively (such as zsh), it's even simpler: md5sum **/* 2>/dev/null – Joost Jul 12 '19 at 16:45
  • if the process will take a while, you can run it through pv to track progress. use it in line mode w/ pv -l: find . -type f -exec md5sum '{}' \; | pv -l -s $(find . -type f | wc -l) > ~/md5sum.txt ...just make sure both of your find filters match: the main one and the one in the pv subshell – zamnuts Nov 24 '19 at 19:06
16

A great checksum creation/verification program is rhash. It creates even SFV compatible files, and checks them too.

It supports md4, md5, sha1, sha512, crc32 and many many other.

Moreover it can do recursive creation (-r option) like md5deep or sha1deep.

Last but not least you can format the output of the checksum file; for example:

rhash --md5 -p '%h,%p\n' -r /home/

outputs a CSV file including the full path of files recursively starting with the /home directory.

I find extremely useful even the -e option rename files by inserting crc32 sum into name.

You can change "md5sum" with "rhash" in the PhoenixNL72 examples.

2
  • 1
    What exactly does '/home/' do here? This tool looks pretty good. – Andy Ibanez Nov 11 '17 at 0:49
  • I think it's an error, it certainly errors for me. The -p is the format for the output. I'll correct it. – pbhj Jul 19 '18 at 18:48
7

Here are two more extensive examples:

  1. Create an md5 file in each directory which doesn't already have one, with absolute paths:

    find "$PWD" -type d | sort | while read dir; do [ ! -f "${dir}"/@md5Sum.md5 ] && echo "Processing " "${dir}" || echo "Skipped " "${dir}" " @md5Sum.md5 already present" ; [ ! -f "${dir}"/@md5Sum.md5 ] &&  md5sum "${dir}"/* > "${dir}"/@md5Sum.md5 ; chmod a=r "${dir}"/@md5Sum.md5;done 
    
  2. Create an md5 file in each folder which doesn't already have one: no paths, only filenames:

    find "$PWD" -type d | sort | while read dir; do cd "${dir}"; [ ! -f @md5Sum.md5 ] && echo "Processing " "${dir}" || echo "Skipped " "${dir}" " @md5Sum.md5 allready present" ; [ ! -f @md5Sum.md5 ] &&  md5sum * > @md5Sum.md5 ; chmod a=r "${dir}"/@md5Sum.md5 ;done 
    

What differs between 1 and 2 is the way the files are presented in the resulting md5 file.

The commands do the following:

  1. Build a list of directory names for the current folder. (Tree)
  2. Sort the folder list.
  3. Check in each directory if the file @md5sum.md5 exists. Output Skipped if it exists, output Processing if it doesn't exist.
  4. If the @md5Sum.md5 file doesn't exist, md5Sum will generate one with the checksums of all the files in the folder. 5) Set the generated @md5Sum.md5 file to read only.

The output of this entire script can be redirected to a file (.....;done > test.log) or piped to another program (like grep). The output will only tell you which directories where skipped and which have been processed.

After a successful run, you will end up with an @md5Sum.md5 file in each subdirectory of your current directory

I named the file @md5Sum.md5 so it'll get listed at the top of the directory in a samba share.

Verifying all @md5Sum.md5 files can be done by the next commands:

find "$PWD" -name @md5Sum.md5 | sort | while read file; do cd "${file%/*}"; md5sum -c @md5Sum.md5; done > checklog.txt

Afterwards you can grep the checklog.txt using grep -v OK to get a list of all files that differ.

To regenerate an @md5Sum.md5 in a specific directory, when you changed or added files for instance, either delete the @md5Sum.md5 file or rename it and run the generate command again.

2
  • Command #1 using full paths: md5sum "${dir}"/* is not picking up hidden files starting with dot. Any remedy ? – user14654 Sep 4 '14 at 13:07
  • Late to the party, but since md5sum wants file names (seems uhappy if you give it a directory) it might be easiest to ask find for help: $ md5sum $( find "$dir" -maxdepth 0 -type f ) The "-type f" pulls regular and .hidden files (also, find has many options, but can be super helpful). Omit the "-maxdepth 0" part if you want everything recursively. Note: this is fragile if the filenames have embedded spaces, then you can read up on xargs and -0 ( en.wikipedia.org/wiki/Xargs ) – jgreve Mar 17 '18 at 3:40
3

I hit this issue, and while the solutions above are elegant, I wanted a quick and dirty hack for this situation: 1 directory, with subdirectories one level deep inside it.

So, enter the directory in a shell and run:

md5sum * */* 2>/dev/null > md5sum.md5

This gets all the files in the top level directory, removes the error warning about the sub directories being directories, and then runs md5sums on the subdirectory contents. Advantage: easy to remember, does exactly what it's supposed to do. I always get confused by find syntax and can never remember it off the top of my head, so no need to loop etc, dealing with spaces in directory names, this one liner worked fine. Not a robust powerful solution, no good for > 1 level of subdirectories, but a quick and easy fix for the problem.

2

Here's mine:

time find dirname/|xargs md5sum |tee dirname.md5

It throws errors when it tries to calculate it for a directory, but it's good enough for me.

1
  • Do 2> /dev/null to ignore error due to directory. – Sandeep Deshmukh May 22 '20 at 8:48

Not the answer you're looking for? Browse other questions tagged or ask your own question.