Forgot your password?
typodupeerror

Flawed AMD Chip Can Lead To Data Corruption 203

Posted by Zonk
from the crunchy-mistakes dept.
Brandonski writes "Apparently AMD allowed some flawed chips to slip through their detection grid. The problem affects only a small number of chips and only single core 2.6 and 2.8 GHz CPUs." From the article: "It is believed that the glitch is triggered when the affected chip's FPU is made to loop through a series of memory-fetch, multiplication and addition operations without any condition checks on the result of the calculations. The loop has to run over and over again for long enough to cause localized heating which together with high ambient temperatures could combine to cause the result of the operation to be recorded incorrectly, leading to data corruption."
This discussion has been archived. No new comments can be posted.

Flawed AMD Chip Can Lead To Data Corruption

Comments Filter:
  • I dub thee (Score:2, Funny)

    by Anonymous Coward
    Fetch Div, son of Eff Div, Heir to Count Zero, and Lord of a new generation of digital serfs, soon to be labled as having "emotional problems."
    • Since this is an AMD problem, expect lots of justifications and defensiveness compared to if this was an Intel problem.
      • If you read the article, it sounded like some sort of apologistic approach anyhow: "If star A is in Uranus while star B is in Neptune, and you look at a pr0n site, then there is a chance that the girl's boobies will appear larger than they are" Basically it states that if the thermals are bad, and you cause a hotspot on the CPU, that there MAY be corruption in the data processed. Thats a bit of a stretch. Its not good, but its not major.
  • by ozmanjusri (601766) <aussie_bob@NOspAm.hotmail.com> on Saturday April 29, 2006 @01:50AM (#15226384) Journal
    Hey, I have an AMD 2.8Ghz. Maybe I should stop refresðN9'óI]öR9ù¥Î6ýPoe}+èa(ê{
  • An old problem (Score:5, Informative)

    by AndrewStephens (815287) on Saturday April 29, 2006 @01:53AM (#15226392) Homepage
    Something similar used to happen on very old processors, back in the day. If certain instructions were executed in tight loops, the chips would experience localised heating and eventually malfunction (sometimes with permanent damage).

    I'm too young to remember the details (I think it goes back to the early eighties at least), but perhaps some of the elder gods that lurk around here might be able to supply more details.

    • I used to burn out a lot of abacus beads.
    • by Jerf (17166) on Saturday April 29, 2006 @02:20AM (#15226480) Journal
      Do not meddle in the affairs of the Elder Gods [wikipedia.org], for you are crunchy, and good with ketchup.
    • Re:An old problem (Score:2, Informative)

      by Dadoo (899435)
      If certain instructions were executed in tight loops, the chips would experience localised heating and eventually malfunction (sometimes with permanent damage).

      You're thinking about magnetic cores.

      Whenever you reverse a core's magnetic field, its temperature rises a little. Keep reversing the field fast enough and for a long enough period of time, and the core (or maybe the wires running through it?) will melt, permanently damaging that bit.
      • "Keep reversing the field fast enough and for a long enough period of time, and the core (or maybe the wires running through it?) will melt"

        Shit, so for once reversing the polarity does more harm than good?

    • Re:An old problem (Score:5, Informative)

      by Mister Transistor (259842) on Saturday April 29, 2006 @02:41AM (#15226538) Journal
      You may be referring to the early MC6800 8-bit processors. The first ones had a major problem in that the internal registers were dynamic RAM style memory, and synchronized to the internal state clock. If you halted the processor for an extended period of time, the refresh clock to them ceased and the registers got hot, drew too much current and burned up!

      I'm pretty sure that gave rise to the joke "Halt and Catch Fire"...

      I always figured that if you were to burn out a register from overuse, it would be the carry bit ;)

      Anyway, as to the story at hand, it sounds like this would only ever occur a) to only 3000 processors total - MAYBE, and b) would only ever happen under such an artifically contrived laboratory stress-test/benchmark situation. Any CPU running in a real system would a) have to do other things like service hardware interrupts, and b) wouldn't do something useless like perform a looping calculation without checking to see if it was done periodically. It really sounds like this is a big non-issue in reality.
      • I agree with your comments on the current story. In reality, all modern processors have flaws that only occur in extrememly unlikely circumstances. This one is not any different.
        • Re:An old problem (Score:5, Insightful)

          by Mister Transistor (259842) on Saturday April 29, 2006 @03:05AM (#15226596) Journal
          I'll go you one better - I have formed my own personal postulate/theory/law that:

          No sufficiently complex system can ever be completely bug-free.

          and it's corollary:

          It is impossible to completely test a sufficiently complex system in every possible way to be certain that it's bug-free.

          In that vein, someone once said "Foolproof is impossible because fools are so ingenious", and "As soon as an idiot-proof system is devised, they go and invent a better idiot!"

          • I have formed my own personal postulate/theory/law... and it's corollary: It is impossible to completely test a sufficiently complex system in every possible way to be certain that it's bug-free.

            Along those lines - many years ago, Professor Turing set out to find a test for [among other things] the possible presence of an infinite loop within a computer program.

            Sadly, though, he didn't get very far with that line of inquiry... [google.com]

          • No sufficiently complex system can ever be completely bug-free.

            and it's corollary:

            It is impossible to completely test a sufficiently complex system in every possible way to be certain that it's bug-free.

            This is why we have automated reasoning systems, theorom provers, etc. They allow us to reduce the set of all possible states down to a set of orthogonal equivilence classes, only one example from which need to actually be tested.

            Now, of course, at some point non-ideal physical characteristics can

        • Now, not having a go at the parent post, but if intel was to release a statement like this, the /. community would be having a field day.

          A defect that is known to give incorrect calculations is a serious issue that should be rectified via microcode update or exchange CPU for free (if microcode can't fix it).

          Intel got raked over the coals for the FDIV problem, and so should AMD unless they do the right thing and offer an exchange/free fix so that users get the functional CPU they intended to purchase.

          s

          • Re:An old problem (Score:5, Informative)

            by something_wicked_thi (918168) on Saturday April 29, 2006 @04:14AM (#15226767)
            RTFA. They are offering a free replacement. However, the FDIV bug was overblown. For most people, it didn't matter (few people were using software that required division precise enough to be affected). This bug is even less worrisome. Its effect is, at the moment, completely unobserved in the wild using real world applications. The FDIV bug was apparent to anyone with a calculator.

            I'm not saying AMD should be let off the hook completely, but the bug isn't a big problem, they are offering free replacements, and they publicized it. The FDIV bug was bigger (though still hardly catastrophic), refused (at first) to offer replacements, and they sat on it. The two scenarios are nowhere near similar. Maybe AMD just has more character than Intel, or maybe they were watching in 94/95 when the FDIV bug happened and they've actually learned from Intel's mistakes. Regardless, this whole story is more of a heads-up to concerned buyers than a criticism of AMD.
            • Re:An old problem (Score:4, Insightful)

              by AWhistler (597388) on Saturday April 29, 2006 @08:50AM (#15227345)
              There is a HUGE difference between this AMD problem and the FDIV bug. The FDIV bug, once found, was one of those "1,2, BANG" bugs (do step 1, then step 2 and BANG, the bug is there). With this AMD bug, you have to do the same operation many times before you see the problem, and then the problem is random (only if it overheats enough). Another possible solution to this is to use better heat sinks. This AMD problem isn't 1,2,BANG. Bugs that are of this nature are orders of magnitude harder to find and characterize.

              But you're right, since Intel blundered so badly on their handling of he FDIV bug, everyone else learned from it.
              • What exactly do you mean "blundered badly"?

                It is a textbook case in many MBA programs how _WELL_ Intel handled this.

                They recalled EVERY CPU at their own expense of millions of dollars. Managing the recall, the disposal, the resupply, the competition, AND the PR nightmare was handled so well that this incident has become canon for MBA candidates.

                • Re:An old problem (Score:3, Interesting)

                  by AWhistler (597388)
                  Then you REALLY need to get new MBA textbooks, since the one you have been reading is too politically correct to be useful. Here is a link from the guy who discovered the bug which includes a timeline (I can't believe his FAQ is still online!)...

                  http://www.trnicely.net/pentbug/pentbug.html/ [trnicely.net]

                  Pay close attention to questions 9, 10, and 11. It explains what REALLY happened, and the author's opinions on the matter, which to my memory are quite accurate. How do I know? At the time I owned a Gateway Pent
                • What exactly do you mean "blundered badly"? It is a textbook case in many MBA programs how _WELL_ Intel handled this. They recalled EVERY CPU at their own expense of millions of dollars. Managing the recall, the disposal, the resupply, the competition, AND the PR nightmare was handled so well that this incident has become canon for MBA candidates.

                  My lord, this reinforces just about every stereotype of b-school students I developed while living in Schwab.

                  First Intel refused to replace the chips, except fo

                • Re:An old problem (Score:3, Interesting)

                  by LurkerXXX (667952)
                  What the hell kind of crappy MBA program did you go to? Intel did *NOT* handle it well. I had one of those CPUs. Intel tried to tell me (a scientific researcher) that my computations didn't need that level of FPU accuracy, and that they wouldn't replace it. It was only after we, the users, screamed bloody murder and brought lawsuits that they decided to back down and replace them all.

                  The PR nightmare was *caused* specifically by the way Intel handled the discovery. They thought that they had the righ

      • These flaws only occur in unlikely circumstances, but they will be useful tools when fighting our new computer overlords.
  • Fearmongering? (Score:2, Interesting)

    by zaguar (881743)
    Is it reasonable to be afraid of this. To exploit this, in a way to allow running of arbitary code, you would need a buffer overflow - which is what this AMD weakness is purporting to allow. However, how many are affected? Only a few of the AMD chips, and AMD has only what, 30% of the market. So to code an exploit, you would be writing to a very limited audience, to a point where it is futile. Why not just exploit the latest create.Textrange of WMF exploit in IE/Windows? Much more money in that.
  • loop through a series of memory-fetch, multiplication and addition operations without any condition checks on the result of the calculations

    I've been saying that for ages, check your results, but naah! Them young'uns and their series of memory-fetch, multiplication and addition operations.
  • Uh oh.. (Score:5, Funny)

    by BigZaphod (12942) on Saturday April 29, 2006 @02:11AM (#15226450) Homepage
    Wow! AMD has invented a way to crash an infinite loop! Awesome! Intel? I bet their solution will take twice as long to crash this loop:

    10 PRINT "HELLO WORLD"
    20 GOTO 10

    AMD is always innovating.
    • Wow! AMD has invented a way to crash an infinite loop! Awesome! Intel? I bet their solution will take twice as long to crash this loop:

      10 PRINT "HELLO WORLD"
      20 GOTO 10

      This loop won't crash. Memory fetch, addition and multiplication, remember ? So you'd need something like this:

      10 I = 10
      20 K = I
      30 K = K + 2
      40 K = K * 2
      50 GOTO 20

      • Re:Uh oh.. (Score:3, Interesting)

        by JollyFinn (267972)
        No that won't crash its FLOATING POINT memory fetch, addition and multiplication loop! Then we need to unroll the loop enough to hide the floatingpoint unit latency. So that it stays active.

        10 I = 10.1
        20 K = I
        21 K2 =I
        22 K3= I
        23 K4= I
        30 K = K + 2.1
        40 K = K * 2.1
        50 K2 = K2 + 2.1
        60 K2 = K2 * 2.1
        70 K3 = K3 + 2.1
        80 K3 = K3 * 2.1
        90 K4 = K4 + 2.1
        100 K4 = K4 * 2.1
        50 GOTO 20
  • by reporter (666905) on Saturday April 29, 2006 @02:23AM (#15226491) Homepage
    In 1994, Intel's Pentium processor suffered from a division error [willamette.edu]. Intel handled the problem by initially requiring customers to "prove" that the error caused a serious impact on the customers' lives before Intel would agree to replace the defective chips. Later, after much pressure and lost credibility, Intel agreed to replace all the defective chips without requiring the customer to "prove" his case.

    AMD has a unique opportunity to do the right thing: offering to replace all the defective chips. If AMD does the right thing, then it will only help AMD in its litigation against Intel and in various attempts to increase marketshare. After all, would you not prefer to buy from a reputable company instead of a dishonest, shifty company?

    • by Anonymous Coward on Saturday April 29, 2006 @02:28AM (#15226505)
      "The company is also working with OEMs to identify affected parts and contact customers who could be affected - if they are, they will be offered free replacements."

      forth paragraph in TFA.
    • Later, after much pressure and lost credibility, Intel agreed to replace all the defective chips without requiring the customer to "prove" his case.

      AMD has a unique opportunity to do the right thing: offering to replace all the defective chips. If AMD does the right thing, then it will only help AMD in its litigation against Intel and in various attempts to increase marketshare. After all, would you not prefer to buy from a reputable company instead of a dishonest, shifty company?

      AMD have probably learn

      • CALL ESP (Score:4, Interesting)

        by Myria (562655) on Saturday April 29, 2006 @03:42AM (#15226680)
        Probably the easiest errata to come by is the instruction "CALL ESP" (or "CALL RSP"). On AMD CPUs, "CALL ESP" will jump to the address in ESP, *then* push the return address. However, on Intel CPUs, it will push the return address first, then jump to the value it just pushed. This is, of course, disasterous if you try to use it.

        According to Intel errata documents, this is a bug in the Pentium Pro that has been kept for several generations. The Pentium and below, except the 8086 and 8088, worked correctly with this instruction.

        If you want to differentiate Intel and AMD in your program and don't want to use CPUID, you can set up a test with CALL ESP.

        Melissa
    • Jesus. The things that people attribute to AMD's "moral superiority" here on Slashdot... It's astounding.

      If AMD does "the right thing" it won't be because of a moral high road. It's because Intel already stepped on a similar PR landmine long ago. Learning from your rival's huge mistakes is not worth high praise. It's just common sense.
  • nice! (Score:3, Interesting)

    by B3ryllium (571199) on Saturday April 29, 2006 @02:28AM (#15226506) Homepage
    Wow, that was fast. FreeBSD already has a patch for this.

    Judging from the posting date, I *really* need to be updating my sources more often. :)

    20060419: p7 FreeBSD-SA-06:14.fpu
                    Correct a local information leakage bug affecting AMD FPUs.

    (could be an unrelated correction, I guess, it doesn't provide much more information in /usr/src/UPDATING)
  • by IvyMike (178408) on Saturday April 29, 2006 @02:50AM (#15226554)
    This is different than the Intel bug; that was a logic flaw, where the chip computed a floating point quantity using an incorrect algorithm. This is an implementation error. In fact, the article mentions that they're going to re-spec the parts and they'll be fine. So if you've got a 2.8Ghz part, and you run this loop at 2.8Ghz (within the old spec), it's like you're "overclocking" (because you're actually outside of AMD's new spec). My guess is that if you over-bought your heatsink and got something better than the stock OEM cooling solution, you would be fine even if you ran this loop all day. Yay, arctic silver!
    • Floating point is hopelessly problematic for the average programmer and too many average programmers wrote the programs from Excel to MS Calculator and by any number of other vendors, all of which had "Pentium bugs" reported, that didn't need particular Intel hardware to be reproduced.
      • I have a P90 (one of those that was remarked down to P75 for the market sweet spot, but because it's really a P90, it runs fine at 90MHz) that has some sort of FP bug... it passes the Calculator test, but locks up with certain math-intensive screen savers, like the old After Dark kaleidoscope. It never showed any other symptoms in its 6 years of useful life, so I didn't bother to RMA it.

        I don't consider this as bad as the Sept.1998 batch of K6-2 450Mhz CPUs that could not run certain 32bit code AT ALL (neit
  • by Khith (608295)
    Just imagine if you had one of those Pinnacle chips and accidently pressed @[=g3,8d]\&fbb=-q]/hk%fg followed by delete..
  • I have used Prime95 in the past to identify problematic configurations. It's a tool whose main goal is to find prime numbers, but it can be used as an excellent stress test for the processor and memory units.

    Could Prime95 be used to identify those AMD chips?
    • It seems unlikely. The synthetic benchmark that they describe is four particular FP instructions, repeated several million times. Firstly no compiler would unroll a loop that far. Secondly, the four instructions would never occur in practice because you would need some memory accesses at some point to load/store data. Between the loop control, and the memory accesses, the FP instructions are broken up enough that they don't trigger the failure.
  • by MROD (101561) on Saturday April 29, 2006 @03:46AM (#15226690) Homepage
    Having read a lot about this flaw it's actually amazing that AMD's quality control found the problem in the first place.

    The actions needed to cause the problem to arise are so extreme that they'd never happen in the field. i.e. Loop through tight floating-point only instructions without any comparisons for maybe hours before the error occurs.

    This would *NEVER* happen in the field. Firstly, in any modern OS the process would have been pre-empted long before any problem could occur (causing other instructions to run and hence stopping the overheating). Secondly, no real-world program would ever do this sort of thing as there would always be a comparison in the loop within the timeframe.

    This is a theoretical problem only in the real world, especially as it only affects about 3000 processors in total (it has been quoted). This is why AMD gave it such a low priority. We should just forget about it and move on.
    • Having read a lot about this flaw it's actually amazing that AMD's quality control found the problem in the first place.

      The actions needed to cause the problem to arise are so extreme that they'd never happen in the field.

      This kind of thing is standard practice. If you want to stress test a piece of hardware, you write specialised test code which will consume the maximum amount of power possible, not a real world program. You have to be sure that nobody will be able to write software which will drive

    • This would *NEVER* happen in the field. Firstly, in any modern OS the process would have been pre-empted long before any problem could occur (causing other instructions to run and hence stopping the overheating). Secondly, no real-world program would ever do this sort of thing as there would always be a comparison in the loop within the timeframe.

      I am not so sure. The TFA said millions of instructions and the chips are capable of billions. So with HZ=100 there is room enough for 28e6 instructions to be ex

  • by Bloater (12932) on Saturday April 29, 2006 @04:38AM (#15226828) Homepage Journal
    If you have any interrupts coming in, or your loop has a termination condition. I think you have to have your hardware set to send an interrupt many hours in the future then start an otherwise nonterminating loop.

    So under normal conditions on normal PC hardware, this simply won't happen.
  • ... I just got my pair of 285s! ... well fortunately I don't do a lot of FPU work like that. That and I run cpufreq in "ondemand" mode so I don't care about heat...

    Tom
  • Funny, the ad that appeared on the comments page had some code [falkag.net] P.S. Anyone remember the HCF instruction (halt and catch fire).
  • Although the article specifies 2.6 and 2.8 ghz opterons, I've crashed my Venice core 3000+ socket 754 7 times from online gaming conditions generated by a particilar application (warcraft 3 TFT)

    I thought it was the graphic card at first, but the type of crash I've been experiencing and the difficulty to reproduce it (I generally have to play AT with a pro gamer and go on about a 7 game win streak to get game conditions right for the crash) and it does have to be warm in my room...

    WC3TFT can reproducably cre
    • I think someone's confusing user error/not enough troubleshooting with an almost not reproducable issue. TFA mentions a lot of instructions without enough pause of FPU code to cool down. This isn't your bug if you're playing WC3. WC3 uses TCP/IP. TCP/IP generates interrupts - lots of interrupts. So many interrupts that your FPU has plenty of time to cool down between calculations. There are many handy ways of troubleshooting this issue of yours, and I'd bet you're not going to identify the problem by some s
  • by redelm (54142) on Saturday April 29, 2006 @12:27PM (#15228282) Homepage
    About 7 years ago, I wrote a suite of open-source CPU stress-tests I called `cpuburn` [sbcglobal.net]. Little optimized assember pgms designed to stress different parts of the CPU. `burnK7` does precisely this FPU dot product.

    Of course, I expect AMD's production testing dept to have far better code, since they will devote more job hours to it and know proprietary chip details. Still, different parts of AMD as emailed me several times to thank me because they found the pgms useful. Great.

    But these guys know what they're doing. Heat transfer from the hot multipliers has to be carefully analysed [3D finite element heat transfer analysis]. I suspect something far more mundane, like someone reducing die or slug thickness, or a mfg problem with the die/slug gap or thermal goop.

  • ... because the multiply-add is the basic building block of digital signal processing.

    You are apt to be doing this extensively when processing audio or video streams.

  • Interesting! (Score:2, Interesting)

    by seebs (15766)
    A friend of mine and I can reliably crash some similar-generation AMD chips with a loop setting a region of memory to all zeroes, but not with a loop setting it to 0xaaaaaaaa. The chips just lock up. Takes anywhere from a few seconds (linux) to a few minutes (windows).

I am not now, nor have I ever been, a member of the demigodic party. -- Dennis Ritchie

Working...