SHA-1 Broken

Nanolith writes "From Bruce Schneier's weblog: 'SHA-1 has been broken. Not a reduced-round version. Not a simplified version. The real thing. The research team of Xiaoyun Wang, Yiqun Lisa Yin, and Hongbo Yu (mostly from Shandong University in China) have been quietly circulating a paper announcing their results...'" Note, though, that Schneier also writes "The paper isn't generally available yet. At this point I can't tell if the attack is real, but the paper looks good and this is a reputable research team."

Sigh

And I just got done upgrading from MD5.

Re:Well...

thank god ROT-13 will never be cracked.

Re:Well...

That's nothing. ROT-26 offers the best encryption as of yet!

Re:Well...

I can't read your post, it seems to be encrypted in that new ROT-26 scheme.

Re:Well...

I think ROT-65536 would work even better, especially for Unicode.

Hah!

That is nothing. This post has been encrypted with an unbreakable one-time-pad! TWICE!

better yet--

What someone really ought to do is use ROT-7.5 twice to decrypt ROT-13.

Re:better yet--

7.5? 13? I'm guessing you aren't the one who broke SHA-1 ... :-p

Re:better yet--

Someone found that ROT-6.5 works better.

Re:Sigh

About a month ago, I needed a mechanism for password hashes.

After some research, I decided that SHA1 was more secure than MD5.

So I hunted down some good public domain SHA1 code, read through it, and added it to my code.

Thanks /.!

Re:Sigh

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

A mechanism to find collisions does not affect SHA-1's strength as a password hashing algorithm or its use in a hashed message authentication code. So you'll be just fine.Z

really? well, i'm not the real frymaster. what do you say to that?

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFCEsqV7Kzi+hL3je0RAl7iAJ41SsgjgwMvrS5+1OLLYp pYkXUPOgCgzSQS c42DLVAjebLYs2VTPkT/iIc= =8699 -----END PGP SIGNATURE-----

Re:Sigh

Yes, they found a way to break the hash function. But as the parent said, it does not mean it's suddenly invalid. Sure, the group found a way to break the algorithim, but look at According to TFA a collision can be found in about 2**69 hash operations. That's 590295810358705651712 attempts before they can find a match, as opposed to the 2**80 (1208925819614629174706176) that was expected before the paper. While the paper means it is orders of magnitude less work, it still means a lot of work for the attacker. Lets look at two relevant examples: disc images and passwords. Lets say I have an ISO disk image. I hack it, and want to modify some of the 'junk' bits using their algorithm. I'd still need to perform 590295810358705651712 hash operations on that image. Computing the hash of a disc is a slow operation. That's not something I could do in a day, week, or even a few months. Perhaps if I had a massivly parallel computer available, I could do it, but not as an individual. For a password, hopefully your system would lock the account long before there are that many failed login attempts. However, if your attacker has that kind of resources, you can assume it is feasable for them to find a hash collision. That's really only significant for governments, multi-national organizations, and other major enterprises, but not for most people.

Re:Sigh

You do realize, of course, that the recent preponderance of IRC-controlled botnets and such could easily be applied to a computational challenge such as this?

Imagine tens of thousands of way-overpowered virus-infected 3Ghz Dell machines chewing threw the data?

Then imagine a beowulf cluster of those.

Re:Sigh

OK, and then let's do some math.

Let's say you have 2^20 (1048576) machines. Let's say each can do 2^20 hashes per second (this is optimistic). Then it will take you 2^29 seconds to find a hash collision-- this is about 17 years.

This doesn't even let you collide with an arbitrary thing-- rather, you can provide something to someone to sign, and have another message that hashes to the same thing.

It is worrisome, though, because perhaps attacks will improve and it'll continue to get cheaper.

Re:Sigh

For a password, hopefully your system would lock the account long before there are that many failed login attempts. However, if your attacker has that kind of resources, you can assume it is feasable for them to find a hash collision. That's really only significant for governments, multi-national organizations, and other major enterprises, but not for most people.

That is not how it works. THey are using the birthday paradox, that is why brute force is 2^80 not 2^160. Put simple the birthday paradox finds two pieces of data with the same hash. It does NOT (as so many posts believe) allow you to find a matching hash to a fixed piece of data. (This would take 2^160, perhaps less with the weaknesses discovered but not close to 2^69). Hence this does not allow you to break passwords.

Re:Sigh

Lets say I have an ISO disk image. I hack it, and want to modify some of the 'junk' bits using their algorithm. I'd still need to perform 590295810358705651712 hash operations on that image. Computing the hash of a disc is a slow operation. That's not something I could do in a day, week, or even a few months. Perhaps if I had a massivly parallel computer available, I could do it, but not as an individual.

No need to compute the hash of a whole disc. You can calculate the internal state of SHA-1 after processing the whole image except - lets say - the last kilobyte (you do it ONCE) and find a collision by modifying only this last kilobyte with great chance of succeeding. There are 2^8192 variants of the last kilobyte, but only 2^160 variants of the hash - that's why you'll probably succeed.

Not a problem (yet)

For password hashes this attack shouldn't be a problem, if it is as described in the article. The attack does only one thing: allows an attacker to generate two streams of data which hash to the same value. This is a problem for digital signatures, because somebody can sign one data stream, then distribute another with the same signature. So the signature doesn't guarantee the data has not been modified. However, this attack does not allow an attacker to magically deduce your password from its hash, or even generate another password that would hash to the same value as yours. So you don't need to immediately jump up and replace SHA-1 wherever you use it.

OTOH, this attack indicates that other types of attacks may be found sooner than was previously thought. So it is still a good idea to move away from SHA-1 in the medium to long term. Though it's not entirely clear what you should move to. And it is not certain that more attacks will be found soon.

Re:Not a problem (yet)

Of course you can make

SHA1(data1) == SHA2(data2) where data1!=data2

because SHA1 maps from a large space to 160bits. There WILL be collisions for any maping like that. The question is can you make,

SHA1(data1)==SHA2(data2) && data1.length==data2.length) ?

Can you make the length of the hashed data to be equal?

If you cannot, then most of the signed hashes cannot be compromised anyway.

Re:Not a problem (yet)

Adding data2.length to the check is basically just increasing the number of bits used for the hash (and thus reducing the number of chunks of data that hash to the same value, and increasing the amount of time required to find a collision). However, length turns out to be a very poor thing to hash on (non-linearity is prized in hash functions, and you can't much more linear than the length function). If you're going to set aside more space for a hash value, it's better to use a longer hash or multiple hashes together.

Re:Not a problem (yet)

Even adding the length restriction, you are still mapping from a large space (the space of all binary strings of length n > 160) to a small space (the space of all binary strings of length 160), so there will still be collisions.

--Pat / [e-mail]

Re:Not a problem (yet)

See the birthday paradox [wikipedia.org].

With this technique, you can create 2 files, by modifying both of them, that have the same hash. That does not mean that you can create a file with a given hash.

Re:Not a problem (yet)

You have a bit of a logic flaw in your comment.

Maybe you don't realise where he is coming from.

With a digital signature, you can easily have knowledge of the signed message (input to message digest function) and thus change the message while retaining the signature.

With a hashed password, you don't have access to the password (input to message digest function).

The hashed password would require figuring out the password so as to allow changing it to make the same hash. This requires going the wrong way against this one way hash algorithm. If you were able to do this, then you would not bother generating an equivalent password, because you would know the original.

I think the point is, that the one way nature of SHA-1 might still be strong. Meaning digital signatures are weak, but hashed passwords are not.

There is no logic flaw in his comment.

Both statements are fine -- salt explained

Actually, both statements can coexist. In most password systems, the hash of the password itself is not stored; rather, it is a hash of the password concatenated with a string of random characters.

For example, if my password is "foobar", then the server does not store "8843d7f92416211de9ebb963ff4ce28125932878" as the hash, but perhaps the hash of "foobarDKTUHRAOHL" or "19747e26b86ee7939c046c0171a991926f0e01ae". The salt value "DKTUHRAOHL" is stored on the server and never revealed to anyone. So, even if somebody knows the hash value "19747...e01ae", they cannot come up with another string of characters that hashes to the same value, because even if they could, the value they enter in an attempt to hack my account is appended with "DKTUHRAOHL", rendering (almost certainly) a different hash value.

Now, if they know the salt value, the problem becomes equivalent to finding a string ending with "DKTUHRAOHL" that hashes to "19747...e01ae." However, if someone has gained access to a properly secured server's salt values, you have a large problem on your hands indeed.

Re:Both statements are fine -- salt explained

Actually most implementations store the salt along with the hash, so that you can move passwords from system to system, systems like nis, certain ldap implementations (read: not Active Directory), etc... wouldn't function if it was a per server salt. It is also much better to come up with a new salt for each password. The main purpose for this is to prevent pre-computed hash tables from being effective. Long live LANMAN :)

Re:Not a problem (yet)

Thought by much of the Slashdot community, as general reaction to this article shows. Until today, the prevailing Slashdot wisdom was that MD5 was weak and broken and SHA-1 was strong. Now we know that's not the case. Maybe this is no surprise to your circle of cryptography guru friends, but nobody told me until now.

Re:Not a problem (yet)

What about what OpenBSD is doing? Have multiple hashes per file (MD5, SHA1, etc..) for a given signature.

Don't panic! 'Broken' is not Cracked

After some research, I decided that SHA1 was more secure than MD5.

MD5 was 'broken' in 1995 by Hans Dobbertin who discovered compressor function collisions. It was almost another 10 years before the compressor function collisions were turned into an attack which produced hash collisions.

So there is a serious security problem here but it does not mean that everything that uses SHA-1 is now vulnerable. There are many applications where MD5 is completely adequate. If you have a really good reason to do so and a really good understanding of the security requirements and risks you can use even something like MD2.

Today paul Kocher complained that Microsoft was using MD5 in its anti-spyware to identify known bad software. This is not actually a major problem, much worse would be using MD5 to identify known good software to keep, that is when a collision would bite. For known bad programs well i don't want any variant of the program to run...

But if you are writing an entirely new application then use SHA-256 or SHA-512, more rounds, more bits.

Meanwhile we need to research some new hash functions pronto.

Info on what exactly SHA-1 is ...

For those interested, here is the actual detailed/lengthy FIPS PUB 180-1 from NIST, [nist.gov] as typical, Wikipedia has a nice summary, [wikipedia.org] and the W3 Folks [w3.org] have a short snippet ...

Re:Info on what exactly SHA-1 is ...

So SHA-1 was created by the NSA, and was broken nine years after it was released. Is there any chance that the NSA knew it had a secret weakness, and promoted it for that specific reason?

Re:Info on what exactly SHA-1 is ...

Any encryption scheme that lasts about 10 years has given a pretty good service I would think.

SHA-1 is not an encryption scheme, it's a cryptographic one-way hash function.

There's a significant difference.

Re:Info on what exactly SHA-1 is ...

The concept is not the same.

Encryption is not in any way hashing, and hashing is not in any way encryption.

A one way hash cannot in any way be decrypted, thats why it's called one way. It's physically not possible.

A hash is not used to "protect" the data you are hashing, it's used to identify the data you are hashing. You can take an unhashed value, hash it and compare it with another hashed value to identify that the two original values were very likely the same value

The strength of a hash is simply how likely it is that the two values were the same value, or conversly, how likely it is that they were not the same value.

When two distinct values have the same resultant hash, we call that a "collision". It should be obvious that there are an infinite number of collisions for a fixed-length hash value - pidgeon hole principle shows you that.

And SHA1 is not "broken", not yet, to be "broken" we would have to have a feasible way to generate a string of data that when hashed produces the same hash as an *already known* hash.

If we could generate collisions for KNOWN hashes, in a reasonable time, then we could use those collisions to falsly identify to systems that use a hash of the original (still unknown) value.

Re:Info on what exactly SHA-1 is ...

Not quite.

For quite a few applications the hash is broken even if I cannot easily find a second string with the same hash as one given. Even if I can "only" at will find two strings with the same hash, that is a pretty serious weakness.

I could, for example, create two documents with the same hash, have you sign one, and then claim you signed the other one. Since the hashes are the same your digital signature will be valid for both.

For other applications, like replacing a signed document with another without being detected you're rigth -- that would only work if one could easily find a document with a given hash.

Re:Info on what exactly SHA-1 is ...

DES had a weakness nobody but the NSA knew about, so they recommended changes to it without saying the reasons for them. years later an attack was found against DES, but the NSA changes prevented it from being useful. Why would they change their tune to SHA-1?

Re:Info on what exactly SHA-1 is ...

DES had a weakness nobody but the NSA knew about, so they recommended changes to it without saying the reasons for them. years later an attack was found against DES, but the NSA changes prevented it from being useful. Why would they change their tune to SHA-1?

You know, of course, that the NSA did the same thing with SHA right? The original algorithm submitted was SHA-0, then the NSA recommended an unexplained minor change.

Last August SHA-0 was broken, so their tweak appears to have added about 6 months of extra life for SHA-1.

Re:Info on what exactly SHA-1 is ...

So SHA-1 was created by the NSA, and was broken nine years after it was released. Is there any chance that the NSA knew it had a secret weakness, and promoted it for that specific reason?

I don't know about this, but when SHA (the Secure Hash Algorithm) was submitted as an approved algorithm for government use, the NSA reviewed it and suggested a minor change. That modified algorithm is what we now know as SHA-1. It was a few years before public-sector cryptographers caught on to what the significance of the changes was (I wish I could explain it, but it is beyond me).

Re:Info on what exactly SHA-1 is ...

I don't know about this, but when SHA (the Secure Hash Algorithm) was submitted as an approved algorithm for government use, the NSA reviewed it and suggested a minor change. That modified algorithm is what we now know as SHA-1.

Note quite right. The NSA invented SHA, but then a couple of years later discovered a weakness and made a slight change, to create SHA-1. The older version is now called SHA-0. According to Schneier's report, SHA-0 was broken even worse by the Chinese team, 2^39 work for a collision vs 2^69 for SHA-1. So the NSA's change was important and made a major increase in the strength of the algorithm.

It was a few years before public-sector cryptographers caught on to what the significance of the changes was (I wish I could explain it, but it is beyond me).

They just added a 1 bit rotate in one phase of the algorithm. Without it, SHA tends to keep the bits lined up and there is less mixing.

Re:Info on what exactly SHA-1 is ...

SHA-1 is not used for encryption, it is used for message authentication. Part of the NSA's mandate is to secure government traffic; it would gain little from promoting a broken digest algorithm. It arguably might have an interest in promoting a broken encryption algorithm, but SHA-1 is used for digital signatures.

Re:Well

Had to happen, didn't it?

No algorithm is all-powerful - it only withstands attacks for so long.

No, it didn't. In fact, this is the most important problem in CS. The theory is that there are certainly problems where checking a solution is easy (2 and 3 are unique factors of 6 because it's easy to see that 2*3 == 6) but where the only possible way to find the solution given the answer is to compute the solution for every possible answer.

It's not been proven whether hashing is this type of problem (whether it's NP-complete). Moreover, it's never been proven that there isn't a solution for problems we think are NP.

What's more, it *has* been proven that once we find a solution to an NP-complete problem we'll instantly have solutions for *every* NP-complete problem.

Not necessarily

Many hashing functions operate by simple bit manipulations. There are classes of hash function which use cellular automata instead. I believe there are some which use fast fourier transforms.

In general, we can say that there are infinitely fewer hashes than there are possible data objects you may wish to hash, and therefore there are infinitely many collisions. We can also say that for an N bit hash, at least one collision must occur over a range of (2^N)+1 values for the initial data object.

However, if the collisions occur on a totally cyclic basis, it doesn't matter if there's only ever one within that range. You know where it is, without the bother of looking.

Therefore, the strength of a hash can be measured as a function of two properties:

• The fewer collisions within one complete cycle, the better.
  
• The more random the distance between collisions, the better.
  

Bit operations have tended to be used, because they're fast and they allow some control over these two parameters. Other than that, there is no particular merit in using them.

Cellular automata can produce some excellent one-way functions. Their behaviour can also be far harder to predict, if the algorithm is good. However, they are computationally very expensive and getting a usefully strong algorithm is much harder than with bit manipulations.

Transforms are not generally considered one-way, because 99.9% of the time they are only useful because they are two-way. I've not really looked into how transform operations are used in hashes, but they presumably have some strengths.

(Transforms in cryptography, where you want to go from one domain to another and then back again, would make sense. They would also be useful for encryption modes, for generating the new encryption key for the next block.)

Re:Yeah...

Well, no. Not exactly. SHA-1 is supposed to be a one-way function, meaning that you can't just reverse the operation. So you can't just "crack" it like solving an equation.

I'm not sure if you are talking about retrieving the original file from the hash, but if you are, then you don't understand what hash functions are for. In this case, there are an infinite number of combinations of bytes that have the same SHA-1 hash. The goal is to find one that has the same hash value, regardless of whether it is actually the same file. SHA-1 is not a cipher.

For more info

I'm not sure if this post is news or what, but for more info, click here:

http://www.itl.nist.gov/fipspubs/fip180-1.htm [nist.gov]

Prison.

A lot of companies and products use SHA1 in some form or another. Does this mean that we can arrest and imprison these "researchers" if they ever step foot in America?

Oh great...

Time to change the VPN policies

Time to switch....

... to SHA-2!

Re:Time to switch....

Which *one* of SHA-2?

FYI, SHA-224, SHA-256, SHA-384, and SHA-512 are all referred to as SHA-2.

Re:Time to switch....

FYI, SHA-224, SHA-256, SHA-384, and SHA-512 are all referred to as SHA-2.


Doesn't matter. The only difference is key length. The algorithm is the same.

Time to start a panic

If you don't switch to the newest, latest hashing algorithm, you will die horribly when your corrupted emacs RPM performs malicious code!!! Everyone, delete everything and log off of the Internets now!!! We're all gonna die!!! HELP!!!

Brought to You By

Same group of people that found the MD5 Hash Collision. Self [slashdot.org] references [slashdot.org] and the MD5 paper [iacr.org].

Re:Please explain further.

both papers were (IIRC) generate two datasets X and Y with the same hash Z
the next step up is to, for any data X and hash Z determine a Y which does not equal X which has hash Z. THe ultimate breakage is when you can, for any data X with hash Z and arbitrary data Y generate M which has the property of Y+M has a hash of Z. At this point you can substitute a conrolled and malicious piece of data which can substitute for X.

May be a big deal...

This may be a big deal, because if I understand correctly, SHA-1 is a similiar algorithm to MD5, which is commonly used to uniquely identify files. If that could be cracked using a similiar technique, a better method of hashing files may have to be found.

What a hash is/does

No, that would be one application of a hash (and not a very good one, because someone wanting to mess with it enroute could just re-hash the doctored version and pass on the new hash. What you discribe could be a way to check for accidental errors, though.). A hash is a function that given data gives a smaller amount of data. This smaller amount of data is then also called the hash of the origonal data. A good hash function has the property that if you know the hash for a file, you shouldn't be able to come up with another file that has the same hash without a prohibitive amount of work. A hash function is broken if this property stops holding.

Re:May be a big deal...

if I understand correctly, SHA-1 is a similiar algorithm to MD5, which is commonly used to uniquely identify files

You do not quite understand correctly. MD5 and SHA-1 are hashing algorithms, and as such it is expected (and accepted) that there are collisions. That is, you might find that your /etc/passwd and /bin/ls files have the same MD5 hash. The value in MD5 and other such hashes is that the probability of that happening is so remote that as a first approximation, comparing hashes is just as good as comparing files.

That is, you can either keep a backup copy of your filesystem to compare against or you can keep a list of hashes, and mathematically, all this "break" has demonstrated is that the chances are 1:590295810358705651712 not 1:1208925819614629174706176 of a collision. In other words, don't lose sleep.

Now, for secure cryptographic signatures, the implications are much more unpleasant. It's not the end of the world, but this is that big red light that says: switch to SHA-512 (or something equally secure) ASAP!

Re:Damn it

It's still not really practically breakable unless this is something bigger than what I'm guessing. SHA-0 was broken a few months ago, and MD5 a while before that. What does it mean for you? Not much.

Some attacker would have to be REALLY dedicated to use this vulnerability to harm you, and they would still require hideous amounts of processor time to mount an effective attack. Digests are a quick and easy way to verify that some message or file is correct. If the hash is signed as well, then you can verify the sender, too. When you download something like a Linux ISO, there is often another file on the server containing the hashes of the files, so you can verify that everything downloaded correctly. If you want to make sure that nobody other than a trusted person modified the files, then that trusted person could encrypt the digest with their private key, allowing anybody with their public key to verify that everything's correct.

A person can, with a broken hash, create another ISO file, perhaps with malicious code inserted, that has the same digest, meaning you can no longer trust the signed digest. Let's say that this vulnerability reduces the average time needed to find a collision from 2^48 tries via the Birthday paradox (If this isn't a 96-bit hash, then I really need to get more sleep) to 2^32 tries. That's over 65,000 times faster, but you know why I'm not worried? That's still over 4,000,000,000 ISO files that the attacker would have to try before hitting on one that's got the wanted characteristics and the correct digest to boot, and if it requires equivalent memory usage to its time usage, then I'd expect it to use at least 48 gigabytes of memory to store all of the previous attempted hashes. If it takes 15 seconds to compute one digest, then you're looking at a mere 2,000 processor years to find a vulnerability, compared to the much more comfortable 130,000,000 processor years that it would have required using the brute force method.

Feel better now? If I really got mixed up, and was wrong about the size, then just multiply all the listed times by 2^32, and wake me in 8 trillion AD.

Now what do we use?

With SHA-1 being MD5's replacement after that was broken, which hash function do we use now?

Re:Now what do we use?

Whirlpool has the same hash length as SHA-256 and is based on the Rijndael encryption function, which is currently believed "safe enough". As such, I'm going to say that that is the best bet right now.

The Hashing Function Lounge [terra.com.br] also lists Cellhash, Parallel FFT-Hash , RIPEMD-128, RIPEMD-160, Subhash and Tiger as (so far) unbroken.

Re:And they scoffed at my continued reliance on MD

Finding a single collision after a huge search isn't the same as being able to generate a collision on demand, which is what the SHA-1 breakage apparently purports to be.

US Secure Hash Algorithm 1

SHA-1 Hash Algorithm [ietf.org] and Source Code [cr0.net].

Bittorrent?

Is it time to update bittorrent?
How hard is it going to be for people to provide garbage data with correct SHA-1 hashes to screw up downloads?

Re:Bittorrent?

I don't think it's practical right now, but that doesn't mean it will hold true for too much longer. As it stands right now, BitTorrent files have 2 hashes: each chunk has a hash, and then the file as a whole has its own hash. This means that for a torrent to be perfectly polluted(that is, polluted without anyone knowing), the garbage data needs to fit both hashes, which will be harder, though breaking a chunk hash is enough to kill a torrent swarm, even if users know about it. However, the **AA organizations aren't exactly poor, and as unlikely as it is, they do have the finances to get access to a large computing cluster, which would allow them to cause some damage.

Judging from what's been said about how difficult it is to break SHA-1 even with this discovery, I would think it's fine for now, but a new hash should probably be included with BitTorrent2.

Broken, but not for everything...

One collision in 2**69 operations... that's quite minimal...

Sure, for signatures, it means that you can't trust the algorithm 100% anymore.

But for storing passwords, and other operations where collisions are not important, it doesn't matter much, even if there's another password that can generate the same hash, you still need to brute-force it.

Re:Broken, but not for everything...

You never could trust it 100%! That was the idea! The algorithm gives you a very high probability of authenticity, not any kind of guarantee (unless the original message is shorter than the output of the hash and everyone who hashes it later absolutely knows the length of the original message).

It's an assurance, that's all. The only guarantee is a one-time pad, and Bruce Schneier's website is full of info on why these aren't practical!

So what's the big deal for the rest of us?

I've been reading about hash collisions for the last few years and haven't figured out why this is a crisis problem.

I'm not a cryptographer, just a nerdy engineer, but let me explain my rationale: a hash algorithm takes an arbitrary message and generates a fixed-length signature that has a high probability (10**50 or better for most modern algorithms) of being the original.

Let's assume that your hash algorithm generates a 128-bit hash. Anyone who knows anything about probability can see that is the original message is greater than 128 bits, there MUST be more than one message that will generate the same hash. For long messages, there may be thousands or millions of messages out of a filed of 10**50 (or better) that have the same hash, although many of them will be meaningless garbage.

So SHA-1 has been broken by a group of cryptographers/mathematicians. Does this really mean that they can generate can alter any message in a way that will generate the same hash as the original, thus fooling the math that we use to validate content? No Way! I read Bruce Scheier's Cryptogram every month and he often makes the same argument.

So yes, this means that from a long-term systems security standpoint, we should all move to stronger hashes. Does it mean that SHA-1-based transactions are inherently secure right now?

I think not!

Re:So what's the big deal for the rest of us?

It is still probably difficult (hard to say without looking at the paper) for someone to find a different document with the same hash as a document you create, but it's now not all that hard to find a pair of documents with the same hash. Someone could give you a document to sign, and get your signature on a different document. Also, IIRC for previous work by this group, the attack applies to chosen pairs of documents with sufficient "random" padding; you can search for a padding for each to generate a hash collision.

Essentially, don't sign anything that someone else has given you without changing it in some way, or your signature might also apply to some other document they have chosen.

Re:So what's the big deal for the rest of us?

Essentially, don't sign anything that someone else has given you without changing it in some way, or your signature might also apply to some other document they have chosen.

Right, this is important.

Decent digital signature protocols (as opposed to just the algorithms) require that you hash more than just the document. For example, you might pick a small amount of random data ("salt"), add that to the message, hash the combination and sign that. You then put the salt in the signature packet so that your signature can be verified.

OpenPGP, for example, requires that certain signature subpackets be part of the hash, such as the signature creation time. It probably should require random salt.

Re:So what's the big deal for the rest of us?

Here's the worry.

Let's say someone trusts a digital signature, signed with SHA-1, to the point of allowing money to be predicated on the validity of this signature. If the message is signed and valid, the payer pays the payee $X dollars, where X is some very large amount.

Message #1 is generated and sent. It validates.

The money is paid. At which point the payee produces a second message which hashes the same as the first but claims to be turning down the deal, or modifying the terms of the deal s.t. they don't have to do anything to earn that money, and they claim that's what was actually sent.

This is a problem, since the break apparently allows the construction of two (relatively) arbitrary message sequences that hash to the same value, which is an easier and much different problem from constructing an arbitrary message that hashes the same as a pre-existing message.

Unfortunately the SHA series seems to be suspect

The Hashing Function Lounge [terra.com.br] lists other problems with the SHA functions:

• (R04) V. Rijmen, "Update on SHA-1", accepted for CT-RSA'2005
  
• P. Hawkes, M. Paddon, G. G. Rose, "On Corrective Patterns for the SHA-2 Family", Cryptology ePrint Archive, Report 2004/207 [iacr.org]
  

If this definite break is confirmed, I think we will need to conclude that the entire family is suspect for any genuinely important purpose.

There are a bunch of hashing algorithms on the Hashing Function Lounge that are listed as having no known attacks. At present, the most widespread is Whirlpool. I think it likely that one of these will replace SHA as the hashing function of choice in major cryptographic areas.

I Can See Bruce Now....

Bruce sits at his desk, reading over the encrypted e-mail sent to him about breaking SHA-1, when a loud scream echoes from his office

I JUST SENT OUT MY NEWSLETTER THIS MORNING!

Brought to you also by....

The MD5 crack team....

http://www.md5crk.com/ [archive.org] (wayback archive)

Just do as federal agencies ave started doing...

Check this article [fcw.com]: Federal agencies have been put on notice that National Institute of Standards and Technology officials plan to phase out a widely used cryptographic hash function known as SHA-1 in favor of larger and stronger hash functions such as SHA-256 and SHA-512.

Some background

Applications that would be broken by this are long-lived cryptographic signatures. Indeed, when a document is "signed", usually only a hash of the document is signed. Finding collisions means one can find two different documents with the same signature.

This affects all applications using SHA-1 for signature, that is signed email (whether PKIX or PGP), server certificates (which are signed documents). This should be mitigated by the fact that in order to be really usable in some cases, the collision must also be meaningful. That is if you find a collision to a signed email but if it is meaningless, you won't really be able to use it to spoof an email. It depends on the attack quality whether collisions are "meaningful" or not.

Some applications that should not be broken are the use of SHA-1 for key derivation, i.e. where one uses SHA-1 essentially as the basis of a random function to generate deterministic new keys from a pre-shared key. (I think that's what Schneier meant by HMAC applications.)

Also, some short-lived signatures should still not be realistically breakable in the time that they would need to be for an attack to be successful; short-lived signatures are typically used in protocols such as IPsec or SSL for authentication. Additionally, to mount an attack on some of these protocols an attacker would need to generate a collision involving "unpredictable" data coming from another party, which the attack may or may not allow.

Someone set us up the bomb

... it looks to me the only solution is wipe Jinan city off the map.
Now where did I leave my nukes....

SHA-1 is broken? Arggggh!

Now I know why my site doesn't work anymore. SHA-1 is broken. Digest::SHA1 won't produce any hashes for me anymore, and I tried to debug the issue but couldn't work out what was going on. Thanks for letting us know SHA-1 is broken Slashdot. I wonder when it will be fixed?

Is it that bad?

The article says that 2**69 hash operations are needed to find a collision. If you have a SuperHashOMatic that can do 1 Billion hashing operations per second, thats still an average time of about 18700 years.

In order for the time to be something to be concerned about (~10 years), you would need a machine capable of doing 1.87e12 hashing operations per second. Thats 1.87 TRILLION hashing operations per second.

Ah, but what about distributed computing?

Let's assume that there are 1 billion desktop computers working on this project. Then they must be able to do 1870 hashing operations per second. This is a ridiculously large number for today's implementations (mine gets 100 per second, most could do about twice that).

So is it bad? Somewhat. Further breaks could make it worse.

We should move away from SHA-1. But this isn't not the end of the world.

Cryptographic break =/= practical break

Note that what cryptographers consider a "break" is not necessarily the same as what users consider a break. (Neither is more strict, they are just different criteria for different people).

In this case, the researchers from Shandong University (supposedly) reduced the work required to find a collision from 2**80 to 2**69; this is a major cryptographic result. It is major because SHA-1, as a "cryptographically strong hash", is not supposed to have any attacks better then random. A factor of 2**11 reduction shows SHA-1 to be very far from ideal; and since lots of clever people have tried to show this, the research team should be proud.

Does this mean the bad-guy-of-your-choice can now start forging digital contracts? Not yet - there is no guarantee that the collision will be meaningful (as least their earlier papers didn't show that result). For a forgery to be useful, the forger needs to make the fake message say something useful - may be change the $1 to $1 million, or change the name, or something. A collision at a random place (or a non-sensical string) is essentially useless as a forgery (there may be some interested DOS attacks, but I am talking about outright forgery which is the point of the hash functions).

And lastly, 2**69 (roughly 10**21) is still a big number! Assume that some clever people wrote a super-duper hand-optimized code that does a whole SHA-1 in a micro-second on a late model 4 Ghz PC, that is 10**6 hashes/sec. A grad-student using all the PC's on a campus, say ten thousand, that's another 10**4. This would take 10**11 seconds (or roughly 20K years). Note that for SHA-0, their break is 2**39 operations, which *is* practical - it would take the grad student only a minute, or a single PC a week.

This break is yet *practical* for *most* people. (Would I still use SHA-1? Not in new application, and I make sure that existing applications get changed over eventually.)

Lest I be accused of ignoring the big boys, the equation changes for them. If a Three Letter Agency is willing to invest a lot of money and design some cool chips that has awsome parallelism and everything, then each break may take only a week. For example, assume these chips has a bunch of pipes that can do a hash every nano-second (or 10**9 hash/second). Further, say there are 100 of these pipes per chip, 100 chips per board, 100 board per rack (or 10**6 pipes/rack). Each rack can then do 10**15 hash/sec, With such a magical rack, it would take 10**6 seconds (or just under two weeks) to find a collision. This would cost Some Real Dollars, but is it within the budget of some three letter agency? You bet. Hack, I would be willing to sell you one for under a billion dollar US. On the other hand, for that kind of money, cryptanalysis takes on different textures - why spend a billion to crack SHA-1 when you can buy the right wet-ware unit for a million?

Re:Cryptographic break =/= practical break

Guess who wants to send meaningless data... bittorrent relies on SHA-1 which is I imagine what the moderator was most interested in.

I might be paranoid but it wouldn't be inconceivably difficult for **AA to upload single blocks of corrupted data and destroy every torrent as it streams, they certainly have the resources.

Missing the point...

It's gone from being a billion times easier, to a half a billion times easier, to just simply find the person responsible and beat any necessary data out of them with a baseball bat and/or knife. Which is cheaper? Extensive studying of cryptography, thousands of dollars of computers, and an extremely long waiting time in order to brute-force something? Or just buying plane tickets, a blunt object, looking up the person's address on MapQuest, and having Cousin Luigi pay a friendly visit?

Poly1305-AES

For people interested in secret-key message authentication: There are authenticators that are (1) much faster than HMAC-SHA-1 and (2) guaranteed to be as secure as AES. In particular, check out Poly1305-AES [cr.yp.to]. Public-domain Poly1305-AES software is now online, though it isn't nicely packaged yet; if you're interested in further announcements, join the Poly1305 mailing list.

(This is not meant as a comment on the security of HMAC-SHA-1.)

What do those numbers mean?

I read on one site - in answer to the question "What's the big deal - is 2**69 really all that bad?"

That's 2**11 less operations. Let's say breaking this (2**69 ops) takes the NSA a week. If it had been 2**80, it would have taken 2048 weeks, or 39 years. If it would have taken the NSA (or whomever) a year to break SHA-1 before, it could be broken in 4 hours.

My guess would be it would still take a lot longer than a week - but would now be in the realm of possibility, whereas before it would have been in the lifetime(s) range. However, this is totally a wild-assed-guess, based on the assumption that it was expected to take 100+ years before this to crack.

missing the overall point

What you have to figure is that with any hash thats shorter than the max amount of data, then the possibility of collisions will occur;
figure that if you could represent every possible combination in 128 bits, you would never need to have 129 bits of data.
Because this is not true all hashes will have collisions. However the chances of multiple hashes all having collisions with altered data is 'pretty damn slim'. So therefore the best solution, most likely in the future, and presently is to authenticate messages, identification (ala ssl certificates**) and binaries with multiple hashs known to be reasonably strong. One doesnt need to be a cryptologist to realize that using something like md5, sha256 and like ripemd160, the chances of collision in all 3 hashes are quite slim, and within the range of acceptable risk.

Can someone

Explain to me why this isn't useful for compression?

I know it's next to impossible to create the data from the hash but shouldn't it be theoretically possible?

If the hash reduces the possible files which match it by 99.999% then shouldn't it be possible to send that much less data?

What to use in new apps?

I'm actually working on an app that was going to use SHA-1 for integrity verification. I may just stick with SHA-1 because I'm not terribly familiar with the other options out there in this realm. So ideally, what should new apps use these days? What would be the recommended "safe" algorithm? And can I find a nice, tested C library/code for it? :-)

Important info on crypto hashes

I guess I missed posting this before the bulk of the posts, but maybe it'll help someone.

First: MD* SHA-* etc - they are all basically the SAME algorithm! The are just minor modifications of the same exact thing, so a break in one is a break in all.

Second: Tons and tons of people ask: can't we merge two hashes together and get a stronger one? Yes you can that's EXACTLY what MD* and HA-* DO! They are a combination of different hashes! That's how they work.

So if you really did have a good combo of hashes then just give them a name and use them as a hash - don't bother just plain merging existing ones.

Also, merging say MD5 and SHA-1 is pointless - they are both based on the same hashing code! You are gaining nothing by merging them.

Well whatever it is...

I hope they get it fixed soon.

Now I get the "Use 2 hash algorithms" comments

Ok, if my file consists of the line "Hello World." then I get the following hashes:

770b95bb61d5b0406c135b6e42260580 for MD5

b924c2f360b572e17c971f1b1b667e0732944df7 for SHA-1

Trying to tinker around with the file and make both hashes come out the same as above would presumably be much more difficult than for any single hashing algorithm, and it might very well be nigh impossible. The little light bulb has finally come on. Now I get it. Yeah using two hash algorithms together would probably work nicely. Don't combine the results mathematically, just append the keys together into a big long string. The final MD5+SHA1 hash key for my file would be:

770b95bb61d5b0406c135b6e42260580b924c2f360b572e1 7c 971f1b1b667e0732944df7

I don't know whether this would be stronger than a SHA-2 of equivalent bit length or not, but now I get what some of you have been saying. From a common sense view, it would seem that something like this would be pretty darn tough to crack, because you would have to make two different algorithms compute matching keys for a given dataset.

Metadata

Maybe we should start encoding meta-data along with the hash, so instead of trusting only on the hash to confirm that the message is from who sign it, we would encode along the message, the size, type and whatever characteristic could define the message.

For instance, suppose I sign the message "Hi, I'm Victor", along with the hash it would contain the size (14 bytes), type (English text), encoding (7bits ASCII) and how about the range of codes used in the messages (from U+0027 - U+0074).

A good hash would give a uniformly distributed random hash for the message, so it is safe to assume that even if we could find a collision, it would be highly unprovable that it would satisfy all the meta-data. In some cases it could be provable that this kind of hash is unbreakable, since there is a finite number of messages that satisfy the meta-data (if you could hash all possibilities and verify that there were no collisions you're 100% safe).

Please define "broken"

That's an awfully vague term. We've got an Ethernet hub with a corner knocked off its case, so theoretically you could say it's "broken", but it still works as well as it ever did. A lot of cryptologic results are like that: we know more than we did before about X, but X is not suddenly rendered useless or even worrisomely less strong. Whereas, in the movies, "we broke their code" generally means, "we have the key and can read their secret messages as quickly and easily as they can."

Not quite the end of the world

So the paper says I can find two values, X and Y, such that they hash to the same SHA-1 value. Great. What can I do with this?

As others have pointed out, I can create 2 documents, X and Y, have a target sign one, then substitute the other. His digital signature will be valid for both. Great - it takes only 2^69 attempts to get a collision - I'm sure the chances that the X and Y found will both be valid English documents, one of which I could convince a target to sign, the other allowing me to scam him out of enough money to make the whole ordeal worthwhile.

However, people keep copies of what they sign. Even if I did find a collision, and even if both documents were valid English text, the guy could say "I didn't sign Y - look, my signature is valid for X - he scammed me". Great.

The more likely scenario is someone signing their own document, then claiming it was fraudulent. They could create their own X and Y, sign X that somehow involves another party, then claim they actually signed Y and this other party was the scammer. But they still have to find X and Y in 2^69 steps such that both make logical sense in the English language - no simple task.

This is cool in a theoretical sense, but in a practical sense, its like saying you don't need a million monkeys on a million typewriters typing for a million years to generate Shakespeare; it'll only take 999,999 monkeys on 999,999 typewriters...

Or, to go back to the theoretical world: with processor speeds doubling every 1.5 years, and this team shaving 11 factors of 2 off of the break time, the lifetime of SHA-1 just shortened by about 16.5 years. Not quite the end of the world as we know it.

Step 1: Break SHA-1
Step 2: ?
Step 3: Profit!

unpublished paper reveals unspecified hole

At least they gave the algorithm. If their synopsis is indicative of the paper, they illustrate that SHA-1 has collisions, and collisions can be discovered through the awesomely sophisticated technique of brute force. Pardon me while I dust off my bomb shelter.

Let's wait for the actual paper. If it takes more CPU power to force a collision within a year than the whole of what IBM sells in that year, I think that the hash is doing its job...

Re:Hmm

It's a hashing algorithm - SHA stands for Secure Hashing Algorithm.

Is it so hard to look it up? [wikipedia.org]

Re:Hmm

sha1 and md5 are generally considered so weak that they should only be used to combat error or accidents, not fraud.

Not true. SHA-1 is the hashing algorithm of practically all common security standards. It's found in SSL/TLS, X.509, PGP (the protocol, not the program, so that means GPG also!), S/MIME, etc. In other words... everything. Replacing this is going to suck. :(

Re:Hmm

Relax... it still takes 2^69 tries. That is 590,295,810,358,705,651,712 hash operations. To brute force sha-1 it takes 2**80. This is only 2**11 times faster then a brute force attack... thats 2048 times faster. Its significant but it's not that big of a deal. It is no more significant then if someone with a 2000 node cluster tried to brute force your hash (which is completely feasible...especially for large government agencies like the NSA). In other words, if you were capable of performing 1 trillion (1,000,000,000,000) hash operations per second, it'd still take nearly 19 years for a collision to be found. I assume the NSA can knock that number down to under 24 hours, but thats expected of them. For anyone else in the world, assuming your not being followed by the NSA... and god help you if you are... sha-1 will still be fine and the entire internet security infastructure will not need to be redesigned.
Regards,
Steve

Re:Start recoding

It is official; Netcraft confirms: SHA1 is dying

One more crippling bombshell hit the already beleaguered cryptohash community when IDC confirmed that cryptohash market share has dropped yet again, now down to less than a fraction of 1 percent of all cryptographic algorithms. Coming on the heels of a recent Netcraft survey which plainly states that SHA1 has lost more market share, this news serves to reinforce what we've known all along. SHA1 is collapsing in complete disarray, as fittingly exemplified by failing dead last in the recent Sys Admin comprehensive cryptography test.

You don't need to be a Kreskin [amdest.com] to predict SHA1's future. The hand writing is on the wall: SHA1 faces a bleak future. In fact there won't be any future at all for SHA1 because SHA1 is dying. Things are looking very bad for SHA1. As many of us are already aware, SHA1 continues to lose market share. Red ink flows like a river of blood.

SHA1 is the most endangered of them all, having lost 93% of its core developers. The sudden and unpleasant departures of long time SHA1 developers Jordan Hubbard and Mike Smith only serve to underscore the point more clearly. There can no longer be any doubt: SHA1 is dying.

Let's keep to the facts and look at the numbers.

MD4 leader Theo states that there are 7000 users of MD4. How many users of MD5 are there? Let's see. The number of MD4 versus MD5 posts on Usenet is roughly in ratio of 5 to 1. Therefore there are about 7000/5 = 1400 MD5 users. SHA2 posts on Usenet are about half of the volume of MD5 posts. Therefore there are about 700 users of SHA2. A recent article put SHA1 at about 80 percent of the cryptohash market. Therefore there are (7000+1400+700)*4 = 36400 SHA1 users. This is consistent with the number of SHA1 Usenet posts.

Due to the troubles of Walnut Creek, abysmal sales and so on, SHA1 went out of business and was taken over by RSA who sell another troubled cryptohash. Now RSA is also dead, its corpse turned over to yet another charnel house.

All major surveys show that SHA1 has steadily declined in market share. SHA1 is very sick and its long term survival prospects are very dim. If SHA1 is to survive at all it will be among cryptographic dilettante dabblers. SHA1 continues to decay. Nothing short of a miracle could save it at this point in time. For all practical purposes, SHA1 is dead.

Fact: SHA1 is dying

Re:So the concern is.....

That's not what's been broken. It's impossible to get the cleartext from a hash - that's why it's called one way (there are an infinite number of cleartexts which can generate that hash, so in theory you can get it, but you've got a 1/infinity probability of picking the right one...)

SHA1 is not 'broken' in any real sense. Someone claims to have reduced the collission rate to 1 in 2**69. That's still bloody small. It'd take your PC a couple of thousand years to check the hashes to generate a collission.

Of course if you had a big enough cluster you could get that down to a year or two I guess.

Man in the middle attacks are *not* what this is about.

Re:So the concern is.....

they could concievably glean the cleartext

No. Hashes like SHA-1 are lossy; there is less information in the hash than in the plaintext. Lost information like that cannot be recovered unless just about everything we know from information theory (and thermodynamics) is wrong.