64GB RAM Under 64-bit Linux? 36
gary.flake asks: "My group at NECI is in need of a machine that can address 64GB of ram in a single process. This means we need 64-bit addressing. We'd prefer to go with a Linux solution because all of our development is under Linux. We've spec'ed out some reasonable machines (Dell can do 32GB, and Compaq can do 64GB) but they seem to be lacking in that they can only be loaded up with 4 x 800Mhz Itaniums. We would really like to have more processing power (2 Ghz x 4 would be a dream). Does anyone know of any monster Itanium machines that will meet our needs? (Please, no Alphas). Finding such a beast is harder than you'd think."
huh? (Score:1)
Finding a machine with 4X2Ghz itaniums isn't hard at all - they simply don't exist. If you don't want alphas, 800Mhz is as fast as the ia64 chips go.
It would help if you'd explain a little more about what you're trying to do, or post your Q to the LKML.
You don't have to be subscribed to post to it.
Also, have you considered NUMA or PVM/MPA architectures? Linux supports the ex-sequent-now-ibm NUMA-Q architecture.
Re:huh? (Score:1)
Of course not! The user-visible architecture didn't change, so user-level pointersd are still 32 bits, and you still have the same 32-bit linear address bottleneck. the upshot is that you are still limited to the usual 3 GB per process limit, although you can have lots of those, and lots of page cache as well.
Re:huh? (Score:1)
> not address 64GB in a single process?
I assume you are talking about 32Bit i386
Linux:
No, highmem only raises the bar for the
total amount of memory, the process is still
limited to it's 32bit address space, I think
the maximum abount of userspace addressable
memory is 3GByte (1GByte reserved for kernel).
Your Linux choices are Alpha, SPARC, and POWER. (Score:1, Interesting)
Of course, Intel sacrificed a lot - particularly actual performance - for a high clock rate and thus the perception of speed from unsophisticated buyers.
The closest alternative is IBM's recently announced POWER4 systems, available in 1.3 and 1.6 GHz options. Extremely nice machines, which will blow the doors off a P4 with twice the clock rate, but if you have to ask about the price, the question is "merely profane, or really, really rude?"
Sun make plenty of machines that large, but their processors have always tended to trail in performance. They keep improving, but stay well behind the bleeding edge.
Alpha is the cheapest option, I think, but it's definitely end-of-life.
Additional options are SGI and HP, but you'd want to go with the vendor Unixes, because the Linux ports aren't ready for serious use yet.
Itanic is, to put it politely, not recommended for serious use. The Merced is a dog, and gcc doesn't help.
2GHz not in the near future... (Score:5, Informative)
You're silly, or in a STRANGE NIICHE'. (Score:3, Insightful)
Either you've got something REALLY REALLY strange going on, or you've got an incredibly strange niche' you are programming in. Your best bet is to spend a few brain cycles and figure out a way to do it WITHOUT 64gb of RAM.
However, if you're hell-bent on getting some CPUs that run Linux, and your budget is unlimited, go for a Sun Ultra Enterprise 10000. Other downside? You'll need a minimum of 16 CPUs to go with that 64 GB of ram (unless Sun released higher density ram chips that were certified for the E10k while I wasn't looking, in which case, you'll only need 8 CPUs). Expect to pay $1-2M.
And then if Linux has been ported to the UltraSparc III processor, you might be able to get away with something smaller. But the Sun V880 will only go 8 CPUs at 32GB. You'll need at least a Sun Fire 3800 to do the job of 64GB. Probably will take 4 processors to make the 64gb available.
The biggest memory being put out for PCs right now are 1gb modules (that I've seen). I don't think you're going to find a motherboard with room for 64 memory modules. Or even 32.
Time to rethink what you are doing, or throw lots of money at it. That money, though, would probably be better spent at buying the brainpower to rethink it.
Sorry.
Re:You're silly, or in a STRANGE NIICHE'. (Score:3, Informative)
Re:You're silly, or in a STRANGE NIICHE'. (Score:1)
Re:Pricewatch (Score:1)
Also the 4GB ram there is NOT a single 4GB Dimm, but rather 4x 1G PC100 ECC Memory
-OZ
Re:You're silly, or in a STRANGE NIICHE'. (Score:1)
> * 4 piece kit - 4x1Gb
> 4Gb Kit (4pcs) PC100 ECC Registered w/IBM $1,280 (Buy Now)
The thing I really hate about Pricewatch is how easy it is to think you're comparing the same item, when you are not. (Especially kits which have slightly different things in them, or OEM products which are lesser than the retail kits.)
But I very much admire your work to find those tasty modules!
Re:You're silly, or in a STRANGE NIICHE'. (Score:2)
big iron doesn't use commodity ram (Score:3, Informative)
Sun's ram is similar. Sun uses slightly slower chips, but has many in parallel with some ungodly wide bit paths to and from the memory modules. Again, it's much different than what we have in our desktop PCs.
Keep in mind that these monster machines were designed with some insane requirements and low tolerance for error. As such, they often require much different components to keep everything working. Recall that most DDR-SDRAM based PC mobos up until the past few months shipped with two or three DIMM slots rather than four or eight because of timing and signal issues that hadn't been resolved yet. Big iron systems from Sun, IBM, and Compaq support over 100 CPUs, over 200 GB RAM, and do so without (many) problems. SGI's Origin 3000 supports up to 1 TB of RAM and 512 CPUs per single system (and several such huge machines can be clusterd via multiple 8 gigabit GSN networking connections if a task requires insane compute power).
Re:You're silly, or in a STRANGE NIICHE'. (Score:2)
Maybe instead of a 32-bit machine they should just use a Turing machine? They could use their brainpower to get it working on that just as easily. A commodore-64 hooked up to a mag-tape reader, with a really long tape could do the job.
(sarchasm off)
I guess my point here is that they are a research institute, and it looks like they are doing some strange things there. The probably *can* afford the machine, and it's probably going to be cheaper to throw hardware at the problem that it will be to hack at the problem to make it computable in awkward hardware.
Re:You're silly, or in a STRANGE NIICHE'. (Score:2, Insightful)
Probably a dangerous response to give on Slashdot, but if they've got money to burn, typically speaking, they're not going to be as Linuxcentric as the submitter seems to say they are.
The posting would have gone more like, "We're about to spend $1M on hardware to do this. But if you can tell me some hardware we can do this with on Linux for the cheap, I can convince them!"
Re:You're silly, or in a STRANGE NIICHE'. (Score:2)
"Money to burn" is not the proper way to put it. That implies that they have Gates' bank account. "Adequately and properly funded" might be the right way to put it. If they are working on problems that need a 64 gig address space to solve cleanly, then adequate funding would be enough to buy a machine that would support the research.
Also, these are researchers. Open source usually comes naturally to researchers. After all, they are probably going to try to publish something. Those sort of people would naturally use Linux, not Windows.
Re:You're silly, or in a STRANGE NIICHE'. (Score:2)
Why 64 bit? (Score:1)
The remaining issue is that, under most OS's, a single process is still limitted to 32 bit pointers; that is, 4GB. Even so, you may find that a more commodity system fits your needs quite well, especially as you can pick up eight or ten of these systems at low, low prices in fire sales (where do you think I got mine?).
Re:Why 64 bit? (Score:2)
A good rule of thumb is to never buy an already maxed out system. If you can afford and *need* a machine with 64 GB of RAM today, you very much need one that can handle at least 128 GB for future upgrades.
Use the SPARC, Luke (Score:1)
platform with 64-bit addressing, you should be
looking at SPARCv9. Duh.
Distrubuted platform? (Score:1)
can it not be split into a distributed process?
No Alphas? (Score:1)
It's going to cost (Score:5, Informative)
SunFire 3800
According to the techpub, you'll need both CPU trays (but perhaps with each only half populated with CPUs -- 2 + 2 = 4 CPUs) to house enough memory modules for 64 GB. Base price for a 3800 is $160,000 for two boards. Plus unique RAM (currently $1700 per GB). Keep in mind that the 3800 cannot be upgraded beyond 64 GB or 8 CPUs.
AlphaServer GS80
Looks like you'll need at least 4 CPUs to handle the 64 GB. Base price seems to be around $140,000. Plus unique RAM. Keep in mind that the GS80 cannot be upgraded beyond 64 GB or 8 CPUs.
IBM P660
2 - 8 CPUs. Up to 64 GB RAM. Starts at $66,000. Plus unique RAM. Runs the Linux-Friendly IBM AIX 5L. Keep in mind that the P660 cannot be upgraded beyond 64 GB or 8 CPUs.
SGI Origin 3000
Looking at about $220,000 for one that can accomodate 64 GB. Plus unique RAM (about $900 per GB). The 3000 can be upgraded to 1024 GB (1TB) of RAM and 512 CPUs as a supported configuration. 1 TB and 1024 CPUs unsupported and requiring unsupported OS patches.
Origin 300 would be *much* cheaper, but it only supports 32 GB right now (it will support 64 GB when SGI ships their high density memory modules) and it's nowhere near as expandable or upgradable as the 3000. Origin 300 cannot be upgraded beyond 32 CPUs and will most likely never support more than 64 GB of RAM.
IBM is your best best.
Some jobs do need 64GB RAM (Score:3, Interesting)
What surprised me is the "please, no alphas" comment. A bunch of alphas will have much higher performance than 4 itaniums. Yes, it is end of life'd, but when you are spending hundreds of thousands of dollars on your computer, worrying about the cost of porting to a new architecture when you replace this computer seems penny wise and pound foolish. Besides, as long as you are using the same OS and compiler on both old and new platforms, the "porting" should be mostly just a matter of recompiling.
Re:Some jobs do need 64GB RAM (Score:4, Informative)
Very well put. I do a great deal of work with my university's high performance computing lab and have been exposed to this area for almost four years and continue to be amazed and blown away. By far our most demanding users are from the meteorology research group -- an organization that has been on the forefront of many new weather modeling and weather analysis applications. Their batch jobs range from crazy to insane when it comes to CPU and RAM usage and have allowed for many interesting case studies with single machines, clusters, and different platforms.
Some random observations:
Many of the jobs the meteorology folks run consume 8, 16, or even 32 GB of RAM per instance. Most jobs keep the disk arrays clattering away like a drive stress test. They've tried our cluster and tried our SunFire, but prefer the Origin 3000 (and for lesser jobs, the Origin 2000). Unlike "simple" partitionable tasks such as rendering (or SETI@home), their batch jobs don't take to clustering too well.
Many of our users put preference to total system size and maturity/support of the compilers moreso that the speed of each CPU or the newness of the software. We still have groups that prefer the Origin 2000 over anything else. One user wants us to invest in commercial fortran compilers for the Linux cluster. Many users simply prefer a single large system such as the Origins and the SunFire.
A few users continue to push the envelope. One that comes to mind has been asking for some adjustment to job quotas so that his app can utilize 128 GB of RAM. We're still not sure *what* he wants to do with that ram and are still waiting to hear back from him.
Not everyone uses their machines to surf the web, compile software, or even manipulate 256 megabyte files in Photoshop... for those that handle several terabytes of data every half hour and need to have half a terabyte or more in ram at any given time.... for those people there is truly big iron and it's available today (and yesterday). I belive we had an SGI Challenge XL with 36 R10K/195 CPUs in 1995. Two years ago we got a 128 CPU Origin 2000 and a 64 CPU SunFire. This summer we finally recieved a 256 CPU Origin 3000 with 512 GB of RAM. Big Iron costs a fortune and isn't for your highschool, your ISP, or even for your bank. But for those that *need* it, it's worth every penny.
What do you want from the guy? (Score:3, Insightful)
If somebody would have bothered to check, the man is a Research Scientist Computer Science [nec.com] (lookup Gary Flake) in the NEC research instiute [nec.com], with only 70 more researchers with him in the whole place, and let me guess that he has a lot of money to spend...
Give him a little credit, that what he is doing is probably worth the time and effort he is putting in to it, and be thankfull that you are here to hear about such a project (if he'll only tell us what it's about ...)
More information from the poster (Score:4, Informative)
A suprisingly large number of responders have asked why anyone would need 64GB, and have speculated that we are either lazy, stupid, or have money to burn. While I can't reveal too much, I'll try to give you an approximate idea of the sort of problems that we are working on.
Basically, we are doing two things: graph theoretic algorithms on graphs with hundreds of millions of vertices and billions of edges, and robust classification systems trained on training sets of a similar size.
For our purposes, we need to access these data structures in a deterministic order that cannot be predicted in advanced. We could cache all of our data structures on disk, but the algorithms would require years to complete because of disk seek time. Instead, we are going to try to keep a compressed version of the data in memory. The difference is that the in-memory approach will take minutes as opposed to years.
And if you are thinking that we need a new algorithm, trust me on this, we are using the faster algorithm. Most of the other candidates run in exponential time.
If you want a real clue as to what we're doing, then read this [nec.com]. If you like what you've read and think you can help, then contact me [mailto]. I am hiring.
Re:More information from the poster (Score:2)
If you go with IBM/Sun/Whatever, lease your hardware. You'll save a good deal of money that way.
Also, if it isn't practical to get the configuration you achieve (say, sticking with Intel hardware), you might consider solid state drives as an alternative. Not as fast as RAM, but substantially faster than disk, and will typically saturate an UltraSCSI bus with just one drive.
I realize you are in research. It has just been my experience that when the user community comes knocking at my door with some outrageous demands, chances are, there is a better way that the problem can be solved. I apologize for implying you are a qwack.
Can't wait to read that postscript later.
Re:More information from the poster (Score:1)
QikDATA at a glance.
Protected With internal UPS and automatic back-up to mirrored hard disk drives, QikDATA ensures maximum data protection. QikDATA can be configured without any single point of failure.
Extreme Speed
Takes advantage of the latest 64 bit, 66MHz PCI bus technology and SDRAM speeds - for sustained data throughput of up to 350MB/sec, allowing up to 40,000 I/O transactions per second.
Scalable
Each 1RU sized QikDATA provides between 1GB and 16GB of storage. Up to eight QikDATA units can be combined and spanned to create a single high performance drive of up to 128GB.
Simple integration
Has standard 1RU and PCI card installation. Can be formatted, partitioned or spanned like any other drive.
Easily serviceable
Units are easily accessed for upgrades and part replacement.
A quick search of CDW [cdw.com] shows a price of $31,039.92 for the QikDATA 16GB unit.
Better code, not better hardware? (Score:1)
Perhaps an important question is *why* do you need 64GB of addressable memory? Would it be possible to rewrite some of the code such that you could get by on a few gigabytes? It is possible to parallelize the task and spread it over several smaller systems (cluster?)
I worked for a company which was hitting the limit on Intel processors (1GB addressable per process). In their case, they were reading in huge GIS terrain databases. A better solution was for them to read from disk as needed, rather than load the entire database into RAM. While this may not be possible in your case, it is worth considering.
64 Gb of Ram ? , can't MOSIX workout ? (Score:1)