I need to get the MD5 hash of an online file, and then compare it to a file on the local machine.
How can I do this in bash?
asked Oct 15, 2015 at 10:03
Add a comment
|
5 Answers
You can use curl to fetch the online file:
curl -sL http://www.your.fi/le | md5sum | cut -d ' ' -f 1
To compare against another one, store it in a variable and then proceed:
online_md5="$(curl -sL http://www.your.fi/le | md5sum | cut -d ' ' -f 1)"
local_md5="$(md5sum "$file" | cut -d ' ' -f 1)"
if [ "$online_md5" = "$local_md5" ]; then
echo "hurray, they are equal!"
fi
wget can download to standard output with -O-.
wget http://example.com/some-file.html -O- \
| md5sum \
| cut -f1 -d' ' \
| diff - <(md5sum local-file.html | cut -f1 -d' ')
md5sum appends the file name after the MD5, you can remove it with cut.
-
Could you please explain what the
diff - <(md5sum local-file.html | cut -f1 -d' ')does? Commented Oct 15, 2015 at 10:35 -
@FinlayRoelofs: it runs
diffon the pipe output and output of the command in<(...)process substitution, i.e. it compares the MD5 sums of the downloaded file and the local file as requested.– chorobaCommented Oct 15, 2015 at 11:17
wget -q -O- http://example.com/your_file | md5sum | sed 's:-$:local_file:' | md5sum -c
Replace http://example.com/your_file with the URL of your online file and local_file with the name of your local file
answered Oct 15, 2015 at 10:28
-
-
The output of
md5sumis a line containing the checksum and the file name.md5sum -cchecks that file name for the checksum. thesedcommand replaces the-thatmd5sumuses forstdinwith the name of the local file so themd5sum -cat the end verifies that the local file's checksum is the one of the online file. Commented Oct 15, 2015 at 10:45
You could also just do it directly. Use wget or curl to print the remote file's contents and also print the contents of the local file. Pass both to md5sum and compare the output:
$ md5sum <(wget http://www.exacmple.com/file -O- 2>/dev/null) <(cat localfile)
733f328d8cff7dd89970ec34a70aa14f /dev/fd/63
733f328d8cff7dd89970ec34a70aa14f /dev/fd/62
The first line will be the remote file's md5sum and the second is the local's.
-
are these expected to be the same? for example running:
md5sum <(wget http://stackoverflow.com/opensearch.xml -O- 2>/dev/null) <(cat P:/stack.xml)gives me two different checksums...is there a certain way to download the file?– RafaelCommented Jun 14, 2019 at 17:00 -
@R.M. what file? If you get different checksums then the files are different. What is
P:/stack.xml? Does it maybe have a final newline? Did you save it in a text editor? That would have added a newline.– terdonCommented Jun 14, 2019 at 17:03 -
i downloaded it using
download.file(..., mode = 'wb')in R.– RafaelCommented Jun 14, 2019 at 17:07 -
1@R.M. then it almost certainly added a newline (
\nor, if on Windows as your path suggests,\r\n) to the end of the file. So the content isn't exactly the same because of that final newline.– terdonCommented Jun 14, 2019 at 17:14 -
Hope you don't mind me linking to my question, but I don't think the
\r\nwas the issue--I tried truncating the file by 1 and 2 bytes and still got different checksums than whatwgetreturns. Here's the example: stackoverflow.com/questions/56602271/…– RafaelCommented Jun 14, 2019 at 18:54
Via wget and md5sum and awk as a long one-liner =)
awk 'FNR == NR {a[0]=$1; next} {if (a[0]==$1) {print "match"; exit} {print "no match"}}'\
<(wget -O- -q URL | md5sum)\
<(md5sum local_file)
Example
$ awk 'FNR == NR {a[0]=$1; next} {if (a[0]==$1) {print "match"; exit} {print "no match"}}' <(wget -O- -q http://security.ubuntu.com/ubuntu/pool/main/h/hunspell/libhunspell-1.2-0_1.2.8-6ubuntu1_i386.deb | md5sum) <(md5sum libhunspell-1.2-0_1.2.8-6ubuntu1_i386.deb)
match
$ awk 'FNR == NR {a[0]=$1; next} {if (a[0]==$1) {print "match"; exit} {print "no match"}}' <(wget -O- -q http://security.ubuntu.com/ubuntu/pool/main/h/hunspell/libhunspell-1.2-0_1.2.8-6ubuntu1_i386.deb | md5sum) <(md5sum foo)
no match
