7

Can two different binary files have the same md5 sum? One binary firmware file have different version number and marked as revised, small bug fixed. But both files have same md5 sum, I would assume that revised file can't have the same md5 sum - can this be a mistake?

  • Have you tried yourself from terminal like "md5sum filename" or looking from web for hashes? – al0s Oct 9 '16 at 11:35
  • @al0s I tried md5sum myself from terminal, both md5 are the same. There was no md5 provided by manufacturer. My assumption was that both files are identical, although manufacturer claimed they are different. Now I'm sure they are the same. – minto Oct 9 '16 at 18:24
  • 3
    These answers focus way too much on the cryptographic security aspects of the hash, and far too little on the pragmatic nature of the question. If the hash and size match, you really have the same file. Use diff in this case to be sure. Otherwise sha256sum is getting popular! You can even use it in Windows PowerShell get-filehash. – trognanders Oct 10 '16 at 4:15
  • 3
    Is it possible? Yes. There are an infinite number of binary files, but only a finite number of md5 hashes (since they have fixed size) hence there are infinitely many files that have the same hash. This is true for all hashing functions. Is this the case in your particular situation? Unlikely, but we cannot tell without seeing the actual files... – Bakuriu Oct 10 '16 at 8:16
11

What the existing answers fail to point out is why a collision is deemed to be vanishingly unlikely in this case.

MD5, like any hashing algorithm, was deliberately designed so that a collision won't happen if you just change a handful of characters. You have to change most if not all of them in order to cycle back around to the same hash. That's because the whole point of a hash is to detect single-bit (or few-bit) errors; in this problem domain, you want the smaller changes to definitely trigger a hash change. Flawed though we now know MD5 to be, that property holds to this day.

So, unless the new version of firmware is completely different, and unless you just witnessed a once-in-a-million phenomenon, the odds are huge that you simply received the old version again. Congratulations, because this is the hash-check process working precisely as intended. :)

  • "once-in-a-million" is an incredible understatement. 2^128 is about 10^38.5, so it's closer to one-in-a-duodecillion, an event so unlikely that it's not worth to consider it within the (projected) life time of our solar system. – David Foerster Apr 20 '17 at 17:02
  • @DavidFoerster: "One-in-a-million" is an English idiom, meaning "very rare". It is not intended to be mathematically accurate. – Lightness Races in Orbit Apr 20 '17 at 18:23
  • I'm aware of that but I wanted to highlight that the chance is, in fact, 32 orders of magnitudes lower than the proverbial one-millionth which matters to illustrate the situation. – David Foerster Apr 20 '17 at 21:33
  • +1 =9, 1 more and you get a trinket at 10 :) – WinEunuuchs2Unix Mar 6 '18 at 1:46
18

Of course. MD5's collision vulnerability is well known (see Crypto.SE, Wikipedia). Though it is still a low-probability event, it can be done. However, in your case, I'd suspect a mistake in copying the files.

  • In my case, there was no md5 specified for the FW files, I checked it myself. I think there is a mistake, the files are same, although marked as different revision. – minto Oct 9 '16 at 11:46
  • @minto assuming you didn't make a mistake in downloading the files, it must be a mistake at the server end (or they silently replaced the buggy firmware version as well) – muru Oct 9 '16 at 11:48
  • @mint you can check if the two files are the same or not with diff file1 file2. If the command is silent, then they are the same. – Rmano Oct 9 '16 at 14:23
  • 2
    "I'd suspect a mistake in copying the files" - The chances of this happening by accident are basically 0. If two files have the same md5, either they are the same file, or one are intentionally (maliciously) altered to have the same MD5 as the other. – BlueRaja - Danny Pflughoeft Oct 9 '16 at 21:18
  • 1
    @BlueRaja-DannyPflughoeft I disagree. I have downloaded files and accidentally saved the same file to different filenames (when trying to download them again, because the original download didn't work out for some reason). Human error is also entirely possible on the server side. – muru Oct 10 '16 at 4:13
11

Generally, two files can have the same md5 hash only if their contents are exactly the same. Even a single bit of variation will generate a completely different hash value.

There is one caveat, though: An md5 sum is 128 bits (16 bytes). Since the number of different possible file contents is infinite, and the number of different possible md5 sums is finite, there is a possibility (though small probability in most cases) of collision of hashes. In other words, two different files can produce the same sum when hashed with md5.

Because of this, it's better in some cases to use a higher bit hash (more possible different outputs), to reduce the (already low) probability of an accidental hash collision, and increase the difficulty of creating a deliberate hash collision through brute force.

Examples of higher bit hashes include the SHA-2 family of hashes, especially sha256, sha384, or sha512 (which is the best.) The number after sha indicates the number of bits the corresponding hash algorithm generates.

  • 1
    If one firmware file was revised, and there was some changes done, then md5 sum can't be the same in any case. I used diff -q <file1> <file2> to compare files, and it show the files are identical. – minto Oct 9 '16 at 11:22
  • 10
    "Two files can have the same md5 hash only if their contents are exactly the same, even a single bit of variation would generate a completely different hash value." – This is wrong. Because of the Pigeonhole Principle, there are in fact an infinite number of files which have the same hash. – Jörg W Mittag Oct 9 '16 at 11:36
  • 2
    @JörgWMittag Check the caveat paragraph right after :) – heemayl Oct 9 '16 at 11:37
  • heemayl, @JörgWMittag is right. I've suggested an edit which improves this already very good answer to be precisely accurate and no longer subject to nitpicks. ;) (I hope.) :) – Wildcard Oct 9 '16 at 15:43
  • 2
    For binary (non-text) files, use cmp rather than diff. – waltinator Oct 9 '16 at 18:06
3

As others have said, an MD5 collision is hypothetically possible but extremely implausible (1 in 2^128 is only a 1 in 340,282,366,920,938,463,463,374,607,431,768,211,456 chance), and you most likely have a file-copying error.

I'd recommend doing a byte-by-byte comparison of the two files, using one of the many methods described here: https://superuser.com/questions/125376/how-do-i-compare-binary-files-in-linux.

Or just diff file1 file2 - and unless you get the message "Binary files file1 and file2 differ", the files are the same.

  • Pretty much the same answer as muru. – TheWanderer Oct 9 '16 at 20:46
  • @Zacharee - in my first sentence, I acknowledged those answers which came before. In my second, I directed the reader to a solution. – Dewi Morgan Oct 9 '16 at 21:00
  • 2
    if your answer doesn't really add much more than a link, it would be better as a comment – TheWanderer Oct 9 '16 at 21:57
  • Answer was edited to improve it with further information not given in other answers, in response to these complaints. – Dewi Morgan Dec 12 '18 at 15:32
1

All of the above answers ignore the most important detail:

An MD5 checksum is defined to have 128 bits. That means, there are only 2^128 different MD5 values. How many different firmware images are possible? Well, that depends on how big they are, and it depends on what percentage of random byte sequences could be considered valid firmware. Chances are though, there are more than 2^128 possible firmware images.

A lot more, which means there must be duplicates.

But, the chance of any given firmware image matching a given MD5 checksum is only 1 in 2^128 which is a very small number.

VERY small.

Like, the chance of any two developers accidentally creating different images that have the same MD5 checksum at any time during the existence of human civilization is too small for you to worry about.

That's accidentally. Deliberately is a different question. If you're working for the NSA, then 128 bits is not going to be enough bits of security to satisfy your bosses, and MD5 has known vulnerabilities that make it weaker than 128 bits.

But if you were working for the NSA, then you probably already knew that.

0

Very unlikely but possible. Check the filesize and dates for further information. If the files are different, it would be even more unlikely they would have the same size and hash.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.