Major Linux/Athlon CPU bug discovered
Posted by
chrisd
on Mon Jan 21, 2002 02:49 AM
from the a-bug-a-day-keeps-alan-at-play dept.
from the a-bug-a-day-keeps-alan-at-play dept.
GeorgeFrancisco writes "I recently installed the nVidia drivers so I could play TuxRacer on my Athlon. Problem is it kept inexplicably hanging Linux. Now I know why. The CPU bug affects Athlon/Duron/Athlon MP AGP users. Fortunately there's a way around it, and: "Alan [Cox] is going to try to add some kind of Athlon/AGP CPU bug detection code to the kernel so that it will be able to auto-downgrade to 4K pages when necessary." Read more on the Gentoo Linux site."
This discussion has been archived.
No new comments can be posted.
Major Linux/Athlon CPU bug discovered
|
Log In/Create an Account
| Top
| 402 comments
(Spill at 50!) | Index Only
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
I noticed too (Score:3, Interesting)
Annoyed at something else. (Score:4, Insightful)
What irks me is this: I got hit with this bug. I posted bug reports to Debian, with NVidia, on different forum, report lock-ups in certain open-GL situations. I got generally hand-waving "read the fucking manual" responses.
As the article notes, this isn't just a problem with AMD. It suggests that there's an ongoing problem with troubleshooting and resolving the sorts of issues that desktop users are going to have in Linux. (And "paying for support" would not have resolved much, would it have? The problem is the lack of coordination, not the lack of money.)
Is this the same as the Win2k bug? (Score:4, Interesting)
Re:Is this the same as the Win2k bug? (Score:5, Funny)
Instead of saying "oops, there a hardware bug", they said, "oops, here' a patch for w2k". Looks like none of the kernel developers knew they had to look a w2k bug fixes to find out about hardware bugs.
Re:Is this the same as the Win2k bug? (Score:4, Redundant)
The bad news is that a major Athlon CPU bug has been discovered, and it affects Linux 2.4. Note that this is a bug in the actual CPU itself, and is not a Linux bug. However, it becomes our problem because there are very many semi-broken Athlon/Duron/Athlon MP CPUs out there.
Here are the details. As you may know, x86 systems have traditionally managed memory using 4K pages. However, with the introduction of the Pentium processor, Intel added a new feature called extended paging, which allows 4Mb pages to be used instead. Here's the problem -- many Athlon and Duron CPUs experience memory corruption when extended paging is used in conjunction with AGP. And, this problem hits us because Linux 2.4 kernels compiled with a Pentium-Classic or higher Processor family kernel configuration setting will automatically take advantage of extended paging (for kernel hackers out there, this is the X86_FEATURE_PSE constant defined in include/asm-i386/cpufeature.h.) Fortunately, there is a quick and easy fix for this problem. If you have been experiencing lockups on your Athlon, Duron or Athlon MP system when using AGP video, try passing the mem=nopentium option to your kernel (using GRUB or LILO) at boot-time. This tells Linux to go back to using 4K pages, avoiding this CPU bug. In addition, it should also be possible to avoid this problem by not using AGP on affected systems. As soon as I discovered that this CPU bug existed (which happened, unfortunately, because my CPU has the bug), I informed kernel hacker Andrew Morton of the issue; he put me in touch with Alan Cox. Alan is going to try to add some kind of Athlon/AGP CPU bug detection code to the kernel so that it will be able to auto-downgrade to 4K pages when necessary.
The unfortunate thing about this situation is that AMD and others have known of this bug since September 2000. In fact, AMD's CPG technical marketing division announced this bug on September 21, 2000 in a technical note entitled Microsoft Windows 2000 Patch for AGP Applications on AMD Athlon and AMD Duron Processors (Technical Note TN17 revision 1). And, the kind folks at AMD even created a simple patch for Windows 2000 that disables extended paging by tweaking the registry. However, apparently AMD didn't realize that Linux 2.4 also uses extended paging when the kernel is compiled with a Pentium-Classic or higher Processor family kernel configuration setting. And, it looks like no one in the Linux community noticed that this "Microsoft Windows 2000/AGP Athlon/Duron bug" also applied to Linux 2.4 systems, probably because it was presented by AMD technical marketing as just that -- a Windows 2000-related AGP bug. An unfortunate miscommunication, which has resulted in lots of problems for Athlon, Duron and Athlon MP users. Here's something that's even more unsettling -- consider what kind of Linux users actually use AGP. That's right -- desktop users. And in what area has Linux been struggling? Yes, the desktop. One wonders how many negative desktop Linux experiences have resulted from this unfortunate problem. I don't know if any particular party is to blame for this issue. After all, AMD did prominently announce this bug when it was discovered. But due to an apparently unfortunate series of events, us Linux people never benefitted from this knowledge. But Microsoft Windows 2000 and XP users did. Let's hope that all parties involved can keep things like this from happening in the future.
Re:Is this the same as the Win2k bug? (Score:5, Informative)
Basically, if you run "cat /proc/cpuinfo" and see these:
cpu family: 6
model : 6
stepping : 2
Then you should be safe.
Re:For once Microsoft manged to fix it first (Score:4, Redundant)
Mirror/cache from Google! (Score:3, Redundant)
Thank Google again for this one!
Another mirror/summary here (Score:3, Informative)
The quick answer: (Score:5, Informative)
Boot your system with following option on your kernel command line: "mem=nopentium"
or
Disable AGP in XFree86 config (i.e. Option "NvAGP" "0" in the "Devices" section).
nVidia clued me into the first approach about a week and a half ago. It made my system completely stable. However, there was still some texture flakiness in some OpenGL applications. Since my workstations are number crunchers (and thus Quake FPS don't matter to me), the latter option eliminated both the stability problems and the texture flakiness (at the expense of some graphics speed).
By the way, nVidia mentioned the same issue exists on Win2K / Athlon boxes.
Enjoy,
Kevin
Simple Workaround (Score:3, Redundant)
And yes, this is the same Athlon Windows 2000 AGP bug that was discovered and patched last year with that registry key. They just didn't realize that it also effected Linux until now. I now realize that was the cause of my TuxRacer crashes with my nVidia card on my Athlon computer.
Performance hit? (Score:4, Interesting)
Re:Performance hit? (Score:4, Interesting)
In short, you're better off with 4MB pages if it's stable, but I don't know by how much. I guess some benchmarks would be easy enough to do; e.g. run Q3A with and without the mem= options.
Re:Performance hit? (Score:5, Interesting)
That said, there should be a modest performance hit. Bigger pages can store more data, which results in fewer TLB misses. Hopefully someone will post benchmarks.
Nvidia + AGP + Irongate + Athlon (Score:4, Interesting)
This newly discovered memory corruption with Athlon + AGP, is it contributing to the signal integrity of the Irongate ? Or is it a separate bug ?
Anyway this makes AMD look very bad in my view. There is a bug in the CPU and their chipset screws up my AGP to 1x. Sigh.
Re:Should AMD do the right thing? (Score:4, Informative)
Several processors had self-test instructions known as "HCF". The 6800 family and the 6502 had such instructions. They caused the processor to start fetching consecutive locations, thus continuously incrementing the address bus. Didn't damage the processor, even if you left it running that way. The "Catch Fire" was a figurative description of what was happening on the address bus, nothing more.
On the original NMOS 6502, about 13 of the undefined opcodes had this effect. This was the most common cause of computer lockups if the code went into the weeds.
On some of the later 6800 family members, the test instructions were actually documented, but Motorola's published description did not include any mnemonmic for them.
Re:Should AMD do the right thing? (Score:5, Interesting)
The processors are basicly all their Athlon and Duron processors. For AMD or any chip maker to replace chips with bugs in them is VERY expensive. They already have a low profit margin. Replacing all "defective" Athlon and Duron processors would simply bankrupt AMD. Realisticly, all complex software or hardware has bugs. Bugs in hardware are much more difficult and expensive to fix. The truely significant hardware bugs are usually found early in testing. Other bugs are fixed in software, usually in the system BIOS, but sometimes in the OS code. This isn't something new. It's pretty much always been this way. Why has it been this way? Because no one wants to pay the outlandish prices that would result from trying to make hardware perfect. It costs a tremendous amount of money to reroll a processor. It's not as simple as making a quick code change and recompiling software. THERE WILL ALWAYS BE BUGS IN PROCESSORS! A truely significant bug like the Pentium floating point bug needs to be fixed in the hardware, and that one was even significant enough to deserve a recall of the processor. This bug is simple to work around, and isn't truely a significant problem.
The question you asked in the subject is "Should AMD do the right thing?" The answer is yes, they should correct their Technology Bulletin to actually say what the processor bug is, rather than just say here's a workaround to a bug that effects Win2k.
I'm really surprsed that someone at NVidia didn't pass this on to Linux kernel developers much sooner, since people at that company seem to have been aware of this for some time.
Athlon bug, and NVIDIA drivers (Score:3, Interesting)
The problems I had were frequent lockups with everything X, especially Q3A and Tribes 2. Some experimenting proved what worked and what didn't, and here's what I found:
agpgart never worked worth a damn even with kernel 2.4.17, despite several attempts by me to make it work (I don't maintain it, so I gave up on messing with it). Earlier NVIDIA drivers were less stable, but the latest is great (although it does not support FW, which blows). Tweaking the NVIDIA driver to use SBA and it's own AGP driver instead of agpgart, along with kernel 2.4.14 - 2.4.17 makes for a very stable and fast system. Older kernels just did not work worth a damn whenever I enabled DMA on my IDE drive - they locked every time. These newer kernels don't exhibit this problem, and the NVIDIA driver works nicely with all 3D games as well as 3D development tools like Blender.
My kernels have always been compiled as Athlon kernels as well. The bottom line is: don't blame this bug and/or the NVIDIA driver if your system is unstable and/or slow. There are other things at work, and in my case I seem to have found them all.
- Rohan
How-To: lilo workaround (Score:4, Redundant)
Before the first image= line, insert the line:
append="mem=nopentium"
Does this happen if kernel compiled for K7? (Score:4, Interesting)
By the way, I had to shelve my nVidia card a couple months ago because of this
The equivalent Win2k bug fix (Score:3, Informative)
Funny, I knew something was wrong...
Buggy Features (Score:5, Funny)
Intel: "Not a bug it's erratum."
VIA: "We slowed it down to keep it cool."
Nvidia: "That was a leak! We are not doing public driver beta testing!"
ATI "Who the hell plays Quack3?"
AMD "the patch is here [amd.com]"
Using Test Suites to Validate the Linux Kernel (Score:5, Informative)
Thank you for your attention.
Quake 3 benchmarks (Score:5, Informative)
Without mem=nopentium
FPS = 79.4 (79.4, 79.4, 79.4)
With mem=nopentium
FPS = 79.2 (79.1, 79.3, 79.2)
System tested:
Athlon 850, 384MB RAM, Geforce 1 DDR, VIA KT133 Chipset
Athlon/Duron/K7 optimised 2.4.17 kernel (optimising the kernel above pentium makes very little difference though)
NVidia 1.0-2313 video drivers using agpgart
Mandrake 8.0
Quake 3 settings
Texture depth = 16 bits
Colour depth = 16 bits
Geometric detail = High
Texture detail = High
Dynamic lights = On
Video mode = 1024x768
Looks like there is a difference but it's very slight (0.003%) but my benchmarks aren't very scientific. Either way, if there is an improvement in stability this tradeoff is easily worth it. Here's hoping that you don't run linux just for it's Quake 3 scores [theregister.co.uk] though...
I had a stroll through AMD erratas (Score:3, Interesting)
except model 1 (the very first K7 since it didn't have PSE) are affected,
except the latest revision A5 (cpuid 662) of the Athlon XP, i.e. A0/660 and
A2/661 are affected as well (similarly all 64x Thunderbirds etc.).
(there was a model 1, 2, 4 and 6 Athlon, with 6 being XP)
Some or all Durons might be affected too, but I didn't look at that closely.
The above hinges on whether this is the correct bug description, feel free :)
to flame the anonymous coward if this has got nothing to do with it
"16 INVLPG Instruction Does Not Flush Entire Four-Megabyte Page Properly with Certain Linear Addresses
Normal Specified Operation. After executing an INVLPG instruction the TLB should not contain any
translations for any part of the page frame associated with the designated logical address.
Non-conformance. When the logical address designated by the INVLPG instruction is mapped by a 4-MB
page mapping and LA[21] is equal to one it is possible that the TLB will still retain translations after
the instruction has finished executing.
Potential Effect on System. The residual data in the TLB can result in unexpected data access to stale or
invalid pages of memory.
Suggested Workaround. When using the INVLPG instruction in association with a page that is mapped via
a 4-MB page translation, always clear bit 21."
(page 7 from Athlon Model 6 revision sheet [amd.com])
Alternate, faster? workaround (Score:5, Interesting)
As discussed this may have performance implications.
According to the AMD docs, the problem is only when flushing TLB entries with INVLPG and the page is a 4M page, _and_ the virtual address's bit 21 is set (which does not affect the 4M block of memory the address is in - eg: 0x400000 (2^22) vs 0x600000 (2^22|2^21) are both in the second 4M block).
Hence, when invlpg'ing a VA we just need to INVLPG(address&~(1 (leftshift) 21)). This only requires a single ANDL instruction. But we need to distinguish a 4M page first though, so I don't know?
Heck maybe we should just do it the FreeBSD way and recursively map the Pagedir
Any ideas? Will this work?
--JQuirke
Other Hackers did it better . . . (Score:5, Informative)
Maybe it sheds some light on this issue.
grub workaround (Score:3, Redundant)
For good measure, re-install your grub config by running:
Where /dev/hda is your boot disk.
For most PC users with IDE drives, it will
be /dev/hda .
Last, just reboot.
bad form AMD, realy bad form (Score:3, Troll)
And what do we get for it, no real support, write your own fix, no; that we can, and often do. What we got was forgotten, you didn't even tell us. We are used to and demand full disclosure, and in real time. Linix people hang their dirty laundry out in public to give everyone a fair and equal chance at a fix.
We're often treated as a minority because we are, but treat us as a second class minority at our own peril. In short don't ever let the marketing weenies convince you to hide something from us; if we wanted to be treated that way we would use Win/Intel products
Re:NO AMD BASHING (Score:4, Insightful)
If this is a bug in the processor, AMD really should fix it and offer replacement processors to those who need it. If they don't, and they expect you to patch your OS instead, then that definitely shakes my faith in that company. When you're an artist dependent on OpenGL, you can't have problems like this.
And finally...
Why are you worried about running 32-bit code on a 64-bit processor? 64-bit processors are supposed to run 64-bit code. Intel's not marketing 64-bit processors to replace desktop computers (today), they're for servers and high-end graphics with custom code. They don't NEED to run 32-bit code. I hardly think that's a point against Intel, especially considering they don't make it a big secret that 32-bit code runs slower on it.
Re:NO AMD BASHING (Score:5, Informative)
Just as an aside, if you ever deal with ultrasparcs, you'll quickly find that the majority of the code used is 32 bit.
The reason for it is simple; most applications will run slower at 64 bit than at 32 bit. The ultrasparc chips were designed to take this into account. Hell, due to a firmware bug, solaris on my ultra 1 installs as a 32 bit kernel by defualt - and runs no slower because of it (although it can't run 64 bit apps that way). After a firmware patch, it is easy to change to running the 64 bit kernel though.
In all reality, why would most apps need 64 bit integers and whatnot? Most don't, and doing so is a waste of memory. If the processor is designed right, it can handle 32 bit code with no problems whatsoever.
Re:NO AMD BASHING (Score:4, Interesting)
Not because I do big databases or suchlike, but they let you do loads of optimisations that wouldn't otherwise be possible. For example, you can pass around 8-byte structures in a single register, which is damn useful given the lack of available registers in the x86 architecture.
Example: I've recently been coding a large hexagonal grid component. Each point in the grid is indexed by 2 32-bit (x,y) integers. With a 64-bit register, you could put a full co-ordinate into a single register.
Why is this useful? Well, one of my requirements was to be able to manage large sets of co-ordinates (think reachable spaces for an AI). You want to be able to combine sets of co-ordinates, which basically requires merging two lists. In order to merge lists efficiently, you need to sort them. And with the 64-bit representation, you can do this with just one subtraction and one branch rather than a combination of two subtracts
and two branches. This is a definite speedup if you are hand-coding, and possibly an even bigger one if your compiler doesn't inline all the 32-bit code properly.
Other example: 32-bits are large enough for most integer applications (you couldn't enumerate all the people on the plant though....) but they tend to fall down when you multiply, e.g. 100,000 * 100,000 has already blown the 32-bit limit, and neither of those are particularly big numbers. Whenever you start doing a reasonable amount of multiplication, 64-bit becomes useful.
Also, 64-bits is big enough to encode the positions of pieces on a chess board. You can use bitwise logic to analyse and store positions. GNU chess certainly does it this way. I expect a *cosiderable* speedup in the top chess-playing algorithms when 64-bit becomes widespread.
I'm really keen to se 256-bit arrive to be honest, 2^(2^3) has more elegance than 2^(2*3) and it would allow you to store a set of bytes in one register. Would allow some very cool text-processing tricks.
Course, it might never happen - I predict a move towards massively parallel 64-bit computers rather than stonking 256-bit ones as the next major evolution in processor power.
What bloody bug? (Score:5, Funny)
My nephew spends hours Sliding that little penguin arround with that bloody elevator music going, & not once has there been a freeze or lockup, much to my dissapointment.
Re:NO AMD BASHING (Score:3, Interesting)
-----
Oh great, so they make defective processors, but don't worry because they won't recall them! How in the hell does that make them better than Intel?
Think about it -- If you own an affected part a recall is GOOD!