3

I need to get the MD5 hash of an online file, and then compare it to a file on the local machine.

How can I do this in bash?

3

You can use curl to fetch the online file:

curl -sL http://www.your.fi/le | md5sum | cut -d ' ' -f 1

To compare against another one, store in a var and then proceed:

online_md5="$(curl -sL http://www.your.fi/le | md5sum | cut -d ' ' -f 1)"
local_md5="$(md5sum "$file" | cut -d ' ' -f 1)"

if [ "$online_md5" == "$local_md5" ]; then
    echo "hurray, they are equal!"
fi
1

wget can download to standard output with -O-.

 wget http://example.com/some-file.html -O- \
     | md5sum \
     | cut -f1 -d' ' \
     | diff - <(md5sum local-file.html | cut -f1 -d' ')

md5sum appends the file name after the MD5, you can remove it with cut.

  • Could you please explain what the diff - <(md5sum local-file.html | cut -f1 -d' ') does? – Finlay Roelofs Oct 15 '15 at 10:35
  • @FinlayRoelofs: it runs diff on the pipe output and output of the command in <(...) process substitution, i.e. it compares the MD5 sums of the downloaded file and the local file as requested. – choroba Oct 15 '15 at 11:17
1
 wget -q -O- http://example.com/your_file | md5sum | sed 's:-$:local_file:' | md5sum -c

Replace http://example.com/your_file with the URL of your online file and local_file with the name of your local file

  • Could you please explain what the sed does? – Finlay Roelofs Oct 15 '15 at 10:35
  • The output of md5sum is a line containing the checksum and the file name. md5sum -c checks that file name for the checksum. the sed command replaces the - that md5sum uses for stdin with the name of the local file so the md5sum -c at the end verifies that the local file's checksum is the one of the online file. – Florian Diesch Oct 15 '15 at 10:45
1

You could also just do it directly. Use wget or curl to print the remote file's contents and also print the contents of the local file. Pass both to md5sum and compare the output:

$ md5sum <(wget  http://www.exacmple.com/file -O- 2>/dev/null) <(cat localfile) 
733f328d8cff7dd89970ec34a70aa14f  /dev/fd/63
733f328d8cff7dd89970ec34a70aa14f  /dev/fd/62

The first line will be the remote file's md5sum and the second is the local's.

  • are these expected to be the same? for example running: md5sum <(wget http://stackoverflow.com/opensearch.xml -O- 2>/dev/null) <(cat P:/stack.xml) gives me two different checksums...is there a certain way to download the file? – R.M. Jun 14 '19 at 17:00
  • @R.M. what file? If you get different checksums then the files are different. What is P:/stack.xml? Does it maybe have a final newline? Did you save it in a text editor? That would have added a newline. – terdon Jun 14 '19 at 17:03
  • i downloaded it using download.file(..., mode = 'wb') in R. – R.M. Jun 14 '19 at 17:07
  • 1
    @R.M. then it almost certainly added a newline (\n or, if on Windows as your path suggests, \r\n) to the end of the file. So the content isn't exactly the same because of that final newline. – terdon Jun 14 '19 at 17:14
  • Hope you don't mind me linking to my question, but I don't think the \r\n was the issue--I tried truncating the file by 1 and 2 bytes and still got different checksums than what wget returns. Here's the example: stackoverflow.com/questions/56602271/… – R.M. Jun 14 '19 at 18:54
0

Via wget and md5sum and awk as a long one-liner =)

awk 'FNR == NR {a[0]=$1; next} {if (a[0]==$1) {print "match"; exit} {print "no match"}}'\
 <(wget -O- -q URL | md5sum)\
 <(md5sum local_file)

Example

$ awk 'FNR == NR {a[0]=$1; next} {if (a[0]==$1) {print "match"; exit} {print "no match"}}' <(wget -O- -q http://security.ubuntu.com/ubuntu/pool/main/h/hunspell/libhunspell-1.2-0_1.2.8-6ubuntu1_i386.deb | md5sum) <(md5sum libhunspell-1.2-0_1.2.8-6ubuntu1_i386.deb)
match

$ awk 'FNR == NR {a[0]=$1; next} {if (a[0]==$1) {print "match"; exit} {print "no match"}}' <(wget -O- -q http://security.ubuntu.com/ubuntu/pool/main/h/hunspell/libhunspell-1.2-0_1.2.8-6ubuntu1_i386.deb | md5sum) <(md5sum foo) 
no match

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.