10

I am looking for command or utilities for calculating md5,sha1 hash value by one command.
Right now ubuntu has sha1sum and md5sum command for calculating hash value.

2
  • 2
    Why do you want it? Usually you want to be able to verify hashes after generating them. For example, to generate a hash: md5sum hosts. Then, to verify this result: echo "b9adfb2e2022a3a84ba06b55eeb2dc64 hosts" | md5sum --check (should give: hosts: OK)
    – Lekensteyn
    Dec 3, 2013 at 14:29
  • 2
    suppose if want to calculate 'hash value' of 500 GB hard drive ie both sha1 and md5. if i calculate one by one (using sha1sum and md5sum) would take double time. but if could be done with single utilities same as windows software then would be my desired answer.
    – M S Parmar
    Dec 4, 2013 at 12:08

4 Answers 4

10

You can achieve this with some proper bash ninja-fu. :)

You know the procedure to calculate one at a time:

$ echo abc | md5sum
0bee89b07a248e27c83fc3d5951213c1  -
$ echo abc | sha1sum
03cfd743661f07975fa2f1220c5194cbaff48451  -

Edit: as @gertvdijk suggested, and reading the info pages a bit more, this can be done directly with tee and Process Substitution supported by modern shells, without redirects. This way you can pass your data to two processes and one file using tee:

$ echo abc | tee >(md5sum) >(sha1sum) > output.txt

It's also possible to chain if you need more, but you have to take care of STDOUT from all the subprocesses. This will NOT give you the expected result, but mixes the first two checksums together with the data in output.txt:

$ echo abc | tee >(md5sum) >(sha1sum) | tee >(sha256sum) >(sha512sum) > output.txt

If you redirect the checksums to a file inside the substituted processes, you can chain these as you like:

$ echo abc | tee >(md5sum > /tmp/md5.txt) >(sha1sum > /tmp/sha1.txt) | tee >(sha256sum > /tmp/sha256.txt) >(sha512sum > /tmp/sha512.txt) > output.txt

Here's my initial suggestion without process substitution, but which allows chaining/recursive use without mixing the data and the output:

$ echo abc | tee -a /proc/self/fd/2 2> >(sha1sum) > >(md5sum)
0bee89b07a248e27c83fc3d5951213c1  -
03cfd743661f07975fa2f1220c5194cbaff48451  -

The trick here is to use tee, which duplicates the data to STDOUT and a file. We're being clever by telling it to write the data to the file /proc/self/fd/2, which happens to always be the current process' STDERR file descriptor. And with the > >(program) syntax we can redirect each file descriptor to a program's STDIN instead of a file. Just like |, but with more control. > >(md5sum) redirects STDOUT to the md5sum program, while 2> >(sha1sum) redirects STDERR to the sha1sum program.

Note that the order of 2> and > seems to matter, I have to put 2> first on the command line. These are evaluated from right to left, but I'm not sure why this makes a difference.

To do this on a file or a hard drive, you should replace "echo abc" with a cat or a dd, eg:

dd if=/dev/sda bs=8k | tee -a /proc/self/fd/2 2> >(sha1sum) > >(md5sum)

The nifty thing about this is that you can actually recurse and run several at the same time, not just two. The syntax gets hairy, but this works:

echo abc | tee -a /proc/self/fd/2 2> >(tee -a /proc/self/fd/2 2> >(sha256sum) > >(sha384sum) ) > >(sha512sum)

If you want to capture the result and use it in a script, that works too:

A=$(echo abc | tee -a /proc/self/fd/2 2> >(sha1sum) > >(md5sum))

Now $A is a string containing all the output, including newlines. You can parse the values out later too:

echo "checksum=[$(echo "$A" | head -1 | cut -d " " -f 1)]"

I'm not sure you have any guarantees regarding ordering of the output though.

6
  • 2
    +1. tee and clever use of output redirection in the shell is the way to go. This saves a lot of resources, especially when reading large files.
    – gertvdijk
    Dec 4, 2014 at 11:55
  • 2
    By the way, I think you don't need to redirect to stderr to duplicate the output of the stream. The use of a subshell will also do the trick, maintaining stderr. See my example here in a blog post.
    – gertvdijk
    Dec 4, 2014 at 11:59
  • @gertvdijk Right, process substitution is cleaner, and easier to chain (you don't need to recurse). I'll update my response.
    – ketil
    Dec 4, 2014 at 12:18
  • Nice. I'd give you another upvote if I could. :-)
    – gertvdijk
    Dec 4, 2014 at 12:30
  • While these work well for nice small files, you're doubling the efforts and processing time for larger files which I'm looking to avoid...
    – EkriirkE
    Nov 26, 2015 at 15:47
3

Cant help you with command line but I know a GUI tool named as quickhash.

You can download that tool from Quickhash

Description:

A Linux and Windows GUI to enable the rapid selection and subsequent hashing of files (individually or recursively throughout a folder structure) text and (on Linux) disks. Designed for Linux, but also available for Windows. MD5, SHA1, SHA256, SHA512 available. Output copied to clipboard or saved as CSV\HTML file.

0
Here i have find one python script from source which calculate hash values. and also i find some statistics about hash value calculation.

 - `md5sum` takes 00:3:00 min to calculate 4GB USB.
 - `sha2sum` takes 00:3:01 min to calculate 4GB USB.
 - While phython script takes 3:16 min to calculate both MD5 and SHA1.

//Script start from here

def get_custom_checksum(input_file_name):
    from datetime import datetime
    starttime = datetime.now()
    # START: Actual checksum calculation
    from hashlib import md5, sha1, sha224, sha384, sha256, sha512
    #chunk_size = 1 # 1 byte -- NOT RECOMENDED -- USE AT LEAST 1KB. When 1KB takes 1 min to run, 1B takes 19 minutes to run
    #chunk_size = 1024 # 1 KB
    chunk_size = 1048576 # 1024 B * 1024 B = 1048576 B = 1 MB
    file_md5_checksum = md5()
    file_sha1_checksum = sha1()

    try:
        with open(input_file_name, "rb") as f:
            byte = f.read(chunk_size)
            previous_byte = byte
            byte_size = len(byte)
            file_read_iterations = 1
            while byte:
                file_md5_checksum.update(byte)
                file_sha1_checksum.update(byte)               
                previous_byte = byte
                byte = f.read(chunk_size)
                byte_size += len(byte)
                file_read_iterations += 1
    except IOError:
        print ('File could not be opened: %s' % (input_file_name))
        #exit()
        return
    except:
        raise
    # END: Actual checksum calculation
    # For storage purposes, 1024 bytes = 1 kilobyte
    # For data transfer purposes, 1000 bits = 1 kilobit
    kilo_byte_size = byte_size/1024
    mega_byte_size = kilo_byte_size/1024
    giga_byte_size = mega_byte_size/1024
    bit_size = byte_size*8
    kilo_bit_size = bit_size/1000
    mega_bit_size = kilo_bit_size/1000
    giga_bit_size = mega_bit_size/1000
    last_chunk_size = len(previous_byte)
    stoptime = datetime.now()
    processtime = stoptime-starttime
    custom_checksum_profile = {
        'starttime': starttime,
        'byte_size': byte_size,
        'kilo_byte_size': kilo_byte_size,
        'mega_byte_size': mega_byte_size,
        'giga_byte_size': giga_byte_size,
        'bit_size': bit_size,
        'kilo_bit_size': kilo_bit_size,
        'mega_bit_size': mega_bit_size,
        'giga_bit_size': giga_bit_size,
        'file_read_iterations': file_read_iterations,
        'last_chunk_size': last_chunk_size,
        'md5_checksum': file_md5_checksum.hexdigest(),
        'sha1_checksum': file_sha1_checksum.hexdigest(),        
        'stoptime': stoptime,
        'processtime': processtime,
        }
    return custom_checksum_profile



def print_custom_checksum(input_file_name):
    custom_checksum_profile = get_custom_checksum(input_file_name)
    try:
        print 'Start Time ::', custom_checksum_profile['starttime']

custom_checksum_profile['file_read_iterations']) # print ('Last Chunk (bytes) :', custom_checksum_profile['last_chunk_size']) print 'MD5 ::', custom_checksum_profile['md5_checksum'] print 'SHA1 ::', custom_checksum_profile['sha1_checksum'] print 'Stop Time ::', custom_checksum_profile['stoptime'] print 'Processing Time ::', custom_checksum_profile['processtime'] except TypeError: # 'NoneType' object is not subscriptable --- basically this should happen when the input file could not be opened #raise pass # csv output

import argparse
script_version='0.0.2'
parser = argparse.ArgumentParser(description='Determine and print various checksums of an input file and its size. Supported checksums are MD5, SHA1, SHA224, SHA256, SHA384, and SHA512.', version=script_version)
parser.add_argument('-f', '--file', metavar='in-file', action='store', dest='file_name', type=str, required=True, help='Name of file for which the checksum needs to be calculated')
args = parser.parse_args()
print 'Processing File ::', args.file_name
print_custom_checksum(args.file_name)
0

You don't need "bashisms" (aka process substitution) or in the case you don't have bash but only coreutils GNU tools, you can do it with tee and named pipes.

All in one line:

$ mkfifo sha256; sha256sum sha256 & pv ~/bigfile | tee sha256 | md5sum; rm sha256
[1] 7321
15,5GiB 0:02:57 [89,3MiB/s] [===========================================>] 100%            
02483696f9e698e1cc573bd0154a43e6  -
$ 9a5f63266b2bf13cb09953cd000837dd2b66ce8e5e2945369927b5ec4e324f42  sha256

Note that I am using pv here to display the progress meter (it was a big file: 15.5GB), but a simple cat would work.

Separate steps:

  • Create the fifo
  • Run one of the checksums on the fifo
  • On another shell 'cat' (or 'pv') the file 'tee-ing' it both to the fifo and to the other checksum
  • remove the fifo

You can compute how many checksums you like just by adding more fifos and tees. Unlike @ketil answer, it is "regular", you do not need to do something different each second checksum, just stack them!

You can make it simpler with pee (see here), but this utility is in moreutils that probably does not come pre-installed on your distribution unlike coreutils.

If you want results in a file instead of on the screen just add redirections like that, and it stays "regular" with many checksums

mkfifo sha256; rm -f sums.txt; sha256sum sha256 >>sums.txt & pv ~/bigfile | tee sha256 | md5sum >>sums.txt; rm sha256

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .