SHA-1 Collisions for Meaningful Messages

mrogers writes "Following on the heels of last year's collision search attack against SHA-1, researchers at the Crypto 2006 conference have announced a new attack that allows the attacker to choose part of the colliding messages. "Using the new method, it is possible, for example, to produce two HTML documents with a long nonsense part after the closing </html> tag, which, despite slight differences in the HTML part, thanks to the adapted appendage have the same hash value." A similar attack against MD5 was announced last year."

Re:Not like if it was AES

[rg3]

I'm not an expert so don't take me seriously about it, but: I think hash algorithms are very important when signing things. For example, SHA-1 is the default hash algorithm used in GPG to sign messages. When the first attacks were mentioned I changed that to use RIPEMD-160. If you download something that has a SHA-1 sum to check both correctness and autenticity, it's a problem. Someone could modify the original tarball, for example, introducing a trojan horse, and append some other not useful data to it so the sum matches. I don't know if that's technically feasible as I say, but I imagine the possibilities are not so far. And, furthermore, for me this is another important warning that we should move out of SHA-1 ASAP. BTW, if I recall correctly BitTorrent uses SHA-1 to verify the 256KB chunks. There, having a fixed-size chunk is an advantage for this case, but, as I said, I wouldn't trust SHA-1 much longer. A further step and people could build evil BitTorrent clients that, at least, could corrupt your downloads introducing noise chunks.

Re:Original

[untouchableForce]

It's quite relevant for those using it as a way to verify executables are the way developer had intended. Like the attack last year they're saying it would be possible for someone craft an exe without a virus, generated a checksum for it, get it linked to from major websites (after passing a virus scan of course), and then drop a virus in the end of the file and not have the checksum change. That's the real-world relevance.

Re:This is NOT a big deal

[packetmon]

Even if their test hardware could be accelerated from 33 MHz to 4 GHz, the process would still take 170,000 years. And even if a giant cluster of such machines were used, no collisions would be found within a realistic timeframe of a few years.

The second reason to keep cool is just as important, if not even more so: hackers will have to execute a pre-image attack to manipulate, for instance, a contract that has been digitally signed. In other words, hackers will have to find a second, manipulated contract with the same hash value as the real contract. In principle, the number of operations needed is thus far greater (2160). Indeed, as far as we know all attacks to date have only concerned collisions, and Wang et al. does not change that. There are no known methods to reduce significantly the number of operations needed for pre-image attacks.

Don't you think you're flying off the meter here a bit... Just because a collision was found means truly little. So a garbage laced HTML page was created after the actual HTML closing tag... 1) No one will see what comes after that unless you like viewing the source of a webpage as opposed to an actual page. 2) You should read up on birthday paradoxes. If someone created two similar messages, it would take years for them to figure out how to compute a hash to match. Now in the field of sending out something so so so secure, what makes you think that even if a someone did re-computate a hash to match, that message would be worth anything years down the line. Someone would have to be able to accomplish a collision, re-computate the hash in their new message and send it all within minutes for it to truly be a threat.

Let's look at this scenario... A massive kernel update is made to say Linux... The information is hashed, posted, and everyone is now going to update their Linux boxes... Unless someone is so quick fast to intercept along this path, most are safe unless they choose to verify the hash years down the line (which by then would be worthless). So unless someone can exploit this within minutes (no more than I would guesstimate 36 hours), I see little reason to get all bent out of shape over this...

Re:This is NOT a big deal

[hankwang]

1) No one will see what comes after that unless you like viewing the source of a webpage as opposed to an actual page.

Common web browsers (I just tried Opera, FF, and Lynx) will happily display everything after the closing tag. You would have to put it inside <!-- --> comment delimiters, but then it doesn't matter whether it is before or after the closing tag. Unless the attack doesn't work if the --> has to come at the end, but then you can just omit the closing tag. Only an XHTML-compliant browser would complain. From cursory scanning TFA it is not clear to me what the reason is for mentioning the closing </html.

Re:This is a big deal

[FooBarWidget]

For anyone wanting to use Whirlpool in their apps, here are libraries that you can use:

• Whirlpool library for Ruby. [plan99.net] This is written by me, based on the sample C implementation by the inventors.
• The above library can also be used in C apps. Just copy whirlpool-*.[ch] to your project. See whirlpool-algorithm.h for API.
• The GNU Crypto [gnu.org] library for Java contains a Whirlpool implementation.

Re:Quite simple to check file size also

[wfberg]

I'm not quite sure what your comment means.
As the heise.de article points out; the twins are of equal length - the file size would be the same.
Finding hash twins whereby the chosen one is, oh, let's say 160 bits longer is a degree less sophisticated.

Re:Quite simple to check file size also

[wirelessbuzzers]

I would have thought this not such a big issue for software developers who aren't incompetent.

I don't know the details of this particular attack, but usually attacks on hashes like this produce two documents with the same file size. Certainly the MD5 collisions a couple years ago had the same file size.

NO SHA-1 COLLISIONS HAVE EVER BEEN FOUND

[SiliconEntity]

NO SHA-1 COLLISIONS HAVE EVER BEEN FOUND!

Ahem.

Sorry, my caps lock key got stuck there.

No SHA-1 collisions have ever been found. The first person or group to find one will achieve considerable fame. I say this as an attendee of both last week's Crypto conference and the immediately following hash function workshop.

The work factor estimated for a SHA-1 collision is something over 2^60 cycles. That would put it on par with the biggest calculations that have ever been done (publicly anyway). So far nobody has put together a sufficient effort to achieve a collision.

At the hash function workshop, cryptographer Antoine Joux published a set of recommendations for how such a hash collision effort should be mounted, in order to minimize the damaged from a published collision. The main goal is to make it difficult to take a published collision and use it to create harmful effects in various ways. Hopefully Joux's guidelines will be followed if and when a SHA-1 collision finding project gets started.

Re:Quite simple to check file size also

[edmudama]

From the article:

Using the new method, it is possible, for example, to produce two HTML documents with a long nonsense part after the closing </html>tag, which, despite slight differences in the HTML part, thanks to the adapted appendage have the same hash value.

So it appears that both the original and the new messages need that appendage. This isn't just about adding an appendage to a known, appendage-less document.

Insecure

[zlogic]

SHA-1 was proved to have insecurities years ago. Because of that SHA-2 ("SHA-256", "SHA-384", and "SHA-512") was released back in 2001 as a better version of SHA-1. SHA-2 was tested and no insecurities were found (yet). What's more, SHA-2 is now the official US standard.
Complaining that SHA-1 is insecure is like complaining that Windows 98 is insecure.
Oblig Wikipedia link: http://en.wikipedia.org/wiki/SHA_hash_functions [wikipedia.org]

Re:Add size of file

[evanbd]

And actually reading the hash function descriptions is too complicated a solution for you? Oh, I forgot, you want your karma and most mods will assume you know what you're talking about because they don't read the specs either.

MD5, SHA, and every other hash function I've ever read the spec for appends some zeros followed by the original message size (the zeros are so it comes to an integer number of blocks) as the first thing it does. For exactly this reason.

At a guess, this attack requires that the two files be the same size. (But I haven't actually read TFA.) And attacks that delete or add data but can't manage to correct the file size are the minority anyway.

Re:How about this combination: - Not so good

[Anonymous Coward]

This is broken in O(K 2^(n/2)) time, see Joux's paper "Multicollisions in iterated hash functions".

Re:Insecure

[doublebackslash]

I just want to put a plug out for my good friend whirlpool [wikipedia.org]
Slected as the replacement for SHA-1 years ago by the European NSA, 512 bits long, based on AES.
Who loves ya?

Re:Add size of file

[evanbd]

I'm pretty sure you're confused here. First, note that the padding is a 1 followed by some number of zeros, so no matter what the message actually ends with it is clear what is padding and what is message. That is, in all circumstances adding zeros to the end of the message changes what is being hashed, even when you ignore the file size part. And secondly, the message size as appended onto the message is the *original* message size, *before* any padding occurred. So appending zeros not only changes what the padded file looks like, it also changes the file size string that gets appended after the padding.

I really do think the mathematicians are doing exactly what you guys think they should be.

Re:Why not include message size?

[Anonymous Coward]

Not so: only SHA1 does; MD5 does not. This means that, given two files with the same MD5 has, any fixed file concatenated after both will again produce the same hash. This is how the (MD5-)colliding pair of postscript documents was generated.

Re:This is NOT a big deal

[SanityInAnarchy]

Not true -- XHTML-compliant browsers will only treat it as XHTML if it's sent with a "Content-Type: application/xhtml+xml" header. This is very nice for keeping your page clean -- view it in Firefox with that header, and it'll parse it as XML, and complain whenever you have a problem with your XML, saving you a trip to the even more pedantic W3C validator. Unfortunately, sending that header will have browsers that don't understand XHTML attempt to download the page, rather than displaying it as HTML. Even worse, browsers aren't consistent in the "Accept:" header they send, which is supposed to help with this -- Firefox prefers XHTML and says so, for instance, but while Safari is perfectly capable of receiving XHTML, it just says "Accept: */*", which says it doesn't prefer any type of document -- I may as well send it a PDF (which it would try to download). Even browsers which indicate a preference have to have "*/*" somewhere in there if they want to be able to download files.

So, there's really no good, standard way of detecting whether to send XHTML as XHTML or HTML. I've tried implementing something on my website [slaphack.com], but I doubt that it will really work properly until I break down and attempt to detect specific versions of browsers.

A little information theory

[bwcbwc]

Actually, hashes are difficult to secure for general communications purposes without putting a cap on the size of the transmission. In information-content terms, a collision proof hash is equivalent to a lossless compression algorithm.

A hash will either contain all of the non-redundant information in the original content, or some of the information gets lost during the hash. Non-redundant information being defined in an information-theory sense that a given bit is completely random/unpredictable based on the content of preceding bits.

In order for a hash to be completely collision proof, it has to contain all of the non-redundant information contained in the original file. Otherwise information in the orignal message is lost in the hash. And if information is lost from the original message, that creates a possibility of constructing a message that differs only in the information that is removed by the hash. Only if the original message is reconstructible from the hash (plus possible information contained in the hash algorithm itself) will it be collision-proof. You've either got the information-content, or you don't. And if you don't have the content, you can't validate it.

Re:Quite simple to check file size also

[Stellian]

I have to say, trusting SHA-1 to do what it says on the tin, is not incompetent. Naive, sure, but not incompetent.

Let's not forget this attack is against a reduced 64 round version of SHA-1.
SHA-1 still does what it says on the tin: the best attack known against the 80-round version is 2^63, which is still not practical.