I am looking for command or utilities for calculating md5,sha1 hash value by one command.
Right now ubuntu has sha1sum
and md5sum
command for calculating hash
value.
4 Answers
You can achieve this with some proper bash ninja-fu. :)
You know the procedure to calculate one at a time:
$ echo abc | md5sum
0bee89b07a248e27c83fc3d5951213c1 -
$ echo abc | sha1sum
03cfd743661f07975fa2f1220c5194cbaff48451 -
Edit: as @gertvdijk suggested, and reading the info pages a bit more, this can be done directly with tee and Process Substitution supported by modern shells, without redirects. This way you can pass your data to two processes and one file using tee:
$ echo abc | tee >(md5sum) >(sha1sum) > output.txt
It's also possible to chain if you need more, but you have to take care of STDOUT from all the subprocesses. This will NOT give you the expected result, but mixes the first two checksums together with the data in output.txt:
$ echo abc | tee >(md5sum) >(sha1sum) | tee >(sha256sum) >(sha512sum) > output.txt
If you redirect the checksums to a file inside the substituted processes, you can chain these as you like:
$ echo abc | tee >(md5sum > /tmp/md5.txt) >(sha1sum > /tmp/sha1.txt) | tee >(sha256sum > /tmp/sha256.txt) >(sha512sum > /tmp/sha512.txt) > output.txt
Here's my initial suggestion without process substitution, but which allows chaining/recursive use without mixing the data and the output:
$ echo abc | tee -a /proc/self/fd/2 2> >(sha1sum) > >(md5sum)
0bee89b07a248e27c83fc3d5951213c1 -
03cfd743661f07975fa2f1220c5194cbaff48451 -
The trick here is to use tee
, which duplicates the data to STDOUT and a file. We're being clever by telling it to write the data to the file /proc/self/fd/2, which happens to always be the current process' STDERR file descriptor. And with the > >(program)
syntax we can redirect each file descriptor to a program's STDIN instead of a file. Just like |
, but with more control. > >(md5sum)
redirects STDOUT to the md5sum
program, while 2> >(sha1sum)
redirects STDERR to the sha1sum
program.
Note that the order of 2>
and >
seems to matter, I have to put 2>
first on the command line. These are evaluated from right to left, but I'm not sure why this makes a difference.
To do this on a file or a hard drive, you should replace "echo abc" with a cat or a dd, eg:
dd if=/dev/sda bs=8k | tee -a /proc/self/fd/2 2> >(sha1sum) > >(md5sum)
The nifty thing about this is that you can actually recurse and run several at the same time, not just two. The syntax gets hairy, but this works:
echo abc | tee -a /proc/self/fd/2 2> >(tee -a /proc/self/fd/2 2> >(sha256sum) > >(sha384sum) ) > >(sha512sum)
If you want to capture the result and use it in a script, that works too:
A=$(echo abc | tee -a /proc/self/fd/2 2> >(sha1sum) > >(md5sum))
Now $A
is a string containing all the output, including newlines. You can parse the values out later too:
echo "checksum=[$(echo "$A" | head -1 | cut -d " " -f 1)]"
I'm not sure you have any guarantees regarding ordering of the output though.
-
2+1.
tee
and clever use of output redirection in the shell is the way to go. This saves a lot of resources, especially when reading large files. Dec 4, 2014 at 11:55 -
2
-
@gertvdijk Right, process substitution is cleaner, and easier to chain (you don't need to recurse). I'll update my response.– ketilDec 4, 2014 at 12:18
-
-
While these work well for nice small files, you're doubling the efforts and processing time for larger files which I'm looking to avoid...– EkriirkENov 26, 2015 at 15:47
Cant help you with command line but I know a GUI tool named as quickhash.
You can download that tool from Quickhash
Description:
A Linux and Windows GUI to enable the rapid selection and subsequent hashing of files (individually or recursively throughout a folder structure) text and (on Linux) disks. Designed for Linux, but also available for Windows. MD5, SHA1, SHA256, SHA512 available. Output copied to clipboard or saved as CSV\HTML file.
Here i have find one python script from source which calculate hash values. and also i find some statistics about hash value calculation.
- `md5sum` takes 00:3:00 min to calculate 4GB USB.
- `sha2sum` takes 00:3:01 min to calculate 4GB USB.
- While phython script takes 3:16 min to calculate both MD5 and SHA1.
//Script start from here
def get_custom_checksum(input_file_name):
from datetime import datetime
starttime = datetime.now()
# START: Actual checksum calculation
from hashlib import md5, sha1, sha224, sha384, sha256, sha512
#chunk_size = 1 # 1 byte -- NOT RECOMENDED -- USE AT LEAST 1KB. When 1KB takes 1 min to run, 1B takes 19 minutes to run
#chunk_size = 1024 # 1 KB
chunk_size = 1048576 # 1024 B * 1024 B = 1048576 B = 1 MB
file_md5_checksum = md5()
file_sha1_checksum = sha1()
try:
with open(input_file_name, "rb") as f:
byte = f.read(chunk_size)
previous_byte = byte
byte_size = len(byte)
file_read_iterations = 1
while byte:
file_md5_checksum.update(byte)
file_sha1_checksum.update(byte)
previous_byte = byte
byte = f.read(chunk_size)
byte_size += len(byte)
file_read_iterations += 1
except IOError:
print ('File could not be opened: %s' % (input_file_name))
#exit()
return
except:
raise
# END: Actual checksum calculation
# For storage purposes, 1024 bytes = 1 kilobyte
# For data transfer purposes, 1000 bits = 1 kilobit
kilo_byte_size = byte_size/1024
mega_byte_size = kilo_byte_size/1024
giga_byte_size = mega_byte_size/1024
bit_size = byte_size*8
kilo_bit_size = bit_size/1000
mega_bit_size = kilo_bit_size/1000
giga_bit_size = mega_bit_size/1000
last_chunk_size = len(previous_byte)
stoptime = datetime.now()
processtime = stoptime-starttime
custom_checksum_profile = {
'starttime': starttime,
'byte_size': byte_size,
'kilo_byte_size': kilo_byte_size,
'mega_byte_size': mega_byte_size,
'giga_byte_size': giga_byte_size,
'bit_size': bit_size,
'kilo_bit_size': kilo_bit_size,
'mega_bit_size': mega_bit_size,
'giga_bit_size': giga_bit_size,
'file_read_iterations': file_read_iterations,
'last_chunk_size': last_chunk_size,
'md5_checksum': file_md5_checksum.hexdigest(),
'sha1_checksum': file_sha1_checksum.hexdigest(),
'stoptime': stoptime,
'processtime': processtime,
}
return custom_checksum_profile
def print_custom_checksum(input_file_name):
custom_checksum_profile = get_custom_checksum(input_file_name)
try:
print 'Start Time ::', custom_checksum_profile['starttime']
custom_checksum_profile['file_read_iterations']) # print ('Last Chunk (bytes) :', custom_checksum_profile['last_chunk_size']) print 'MD5 ::', custom_checksum_profile['md5_checksum'] print 'SHA1 ::', custom_checksum_profile['sha1_checksum'] print 'Stop Time ::', custom_checksum_profile['stoptime'] print 'Processing Time ::', custom_checksum_profile['processtime'] except TypeError: # 'NoneType' object is not subscriptable --- basically this should happen when the input file could not be opened #raise pass # csv output
import argparse
script_version='0.0.2'
parser = argparse.ArgumentParser(description='Determine and print various checksums of an input file and its size. Supported checksums are MD5, SHA1, SHA224, SHA256, SHA384, and SHA512.', version=script_version)
parser.add_argument('-f', '--file', metavar='in-file', action='store', dest='file_name', type=str, required=True, help='Name of file for which the checksum needs to be calculated')
args = parser.parse_args()
print 'Processing File ::', args.file_name
print_custom_checksum(args.file_name)
You don't need "bashisms" (aka process substitution) or in the case you don't have bash but only coreutils GNU tools, you can do it with tee and named pipes.
All in one line:
$ mkfifo sha256; sha256sum sha256 & pv ~/bigfile | tee sha256 | md5sum; rm sha256
[1] 7321
15,5GiB 0:02:57 [89,3MiB/s] [===========================================>] 100%
02483696f9e698e1cc573bd0154a43e6 -
$ 9a5f63266b2bf13cb09953cd000837dd2b66ce8e5e2945369927b5ec4e324f42 sha256
Note that I am using pv here to display the progress meter (it was a big file: 15.5GB), but a simple cat would work.
Separate steps:
- Create the fifo
- Run one of the checksums on the fifo
- On another shell 'cat' (or 'pv') the file 'tee-ing' it both to the fifo and to the other checksum
- remove the fifo
You can compute how many checksums you like just by adding more fifos and tees. Unlike @ketil answer, it is "regular", you do not need to do something different each second checksum, just stack them!
You can make it simpler with pee (see here), but this utility is in moreutils that probably does not come pre-installed on your distribution unlike coreutils.
If you want results in a file instead of on the screen just add redirections like that, and it stays "regular" with many checksums
mkfifo sha256; rm -f sums.txt; sha256sum sha256 >>sums.txt & pv ~/bigfile | tee sha256 | md5sum >>sums.txt; rm sha256
md5sum hosts
. Then, to verify this result:echo "b9adfb2e2022a3a84ba06b55eeb2dc64 hosts" | md5sum --check
(should give:hosts: OK
)