NFS In A Disk Write Intensive Environment?

tolldog asks: "At Big Idea Productions, a 3D animation studio, we are looking at ways of improving our network and I/O performance. We have been toying with the idea of loadbalancing the NFS server over multiple machines. We have also thought about buying one large machine with several interfaces. What are the advantages and drawbacks of these methods? Are there other approaches that could work?"

"We will probably be going from a gigabit backbone to gigabit to all of the clients, so network speed will be less of a concern than I/O over NFS. Currently we have a 4 proc Origin serving our filespace with a mixture of SGI workstations and Linux render boxes all mounting the same space. The SGI boxes are mostly reads with a few writes when saving files. The Linux boxes are mostly writes, all over NFS2, with large reads at the begining to get the textures and models loaded into memory."

Well, yeah

[bkosse]

But I was referring to numbers he gave RAID levels in his descriptions

WHY do people keep claiming this? (and #'s wrong)

[bkosse]

First off, your numbers are wrong. RAID 0 is striping (no parity), RAID 1 is mirroring.

RAID 5 does not require the multiple read/write combinations. The parity chunk can be (and are in all but the most horribly broken systems) calculated on the fly and then the system assumes it is done unless the parity chunk comes back wrong.

RAID 4, OTOH, is simply RAID 5 without a dispersed parity. This is, for writes, the *WORST* possible RAID level because it puts pressure on a single disk, and RAID 3 is just as bad.

The absolute fastest "standard", "safe" RAID for reads is RAID 0+1 which is striping and then mirroring the stripe. For writes, it fairs fairly well also, particularly when backed with a hardware controller to help mitigate the need for matching drives in order to get the appropriate performance on writes, which is limited by the slowest drive (its biggest flaw, IMO, other than cost).

RAID 5, OTOH, can be quite effective, even doing "read, modify, write", the trick is to be able to ensure a high probability of having the data from a block already in the cache to prevent the need to read it off the disks again. In fact, it can outperform (yes, you read that right) RAID 0+1 in a write environment when backed by a sufficiently good caching system and if you end up having all transactions be larger than a full block size.

So, to answer the question. Well, if you have a large cache AND hardware controller cards, you can go RAID 5 (1 MB cache per 2-5 GB of disk or so, IME). The only caveat is that you might be working with larger individual files which causes a need to increase the cache per GB.

If you need sheer pedal to the metal performance with fault tolerance, RAID 0+1 looks to be it.

Kinda sounds like roll ur own DFS

[gelfling]

Distributed load balanced NFS features are what AFS and DFS do best and there are far fewer calls <or at least there could be depending on how you set it up> for each I/O. The setup for DCE/DFS can be breathtakingingly complex but once you get there it's pretty much the cat's ass.

Re:Not Linux NFS...

[jfrisby]

Were you using NFSv2 or NFSv3 with Linux? At my company we've had severe performance problems with NFSv2 but have gotten a *huge* performance increase with NFSv3...

Admittedly, we're using Linux for clients and our server is a NetApp box, but if Linux's NFSv2 is screwed up for clients I'd guess it's screwed up for servers...

-JF

NFS ? why ?

[Zurk]

Instead of the ONE HUGE SERVER model where your huge server usually gets reduced to a really big 386SX-16 under heavy loads i'd recommend using a huge server to rsync files to smaller machines which act as local caches. In general, use only 15 or so clients per NFS server. i've seen origin 2000s literally die from NFS serving to large numbers of clients (and these were 192CPU origins). Usually the best way IMHO is to do this : connect small clusters of clients via NFS to cheapo SCSI based PCs (can be dual proc) with (say) 40GB HDDs running (insert favourite free nix). Use rsync to transfer from your huge server to those little boxes when a file request comes in and let the little box act as a temporary file cache effectively. delete files when the little boxes become full. for writing, simply reverse the process. A few shell scripts should do it.

Re:WHY do people keep claiming this? (and #'s wron

[Zurk]

and to give you some numbers - ive seen 90+ megs/sec (thats bytes) sustained transfers from a Sun A5000 series FCAL RAID array with fibrechannel cards using RAID 5. Suns hardware cards have 256MB on controller cache per fibrechannel card. most good arrays will come with fiber channel interfaces to the disks and to the host systems.

Not Linux NFS...

[costas]

I have been administering a couple of high-end Linux clusters, and let me tell you: NFS for distributed writes is a disaster under Linux.

After days and days of testing we eventually came to the conclusion that linux NFS cannot deal with multiple writes to the same server at the same time. We were using quad Xeon IIIs and gigabit networking (cLAN by Giganet) on a dedicated, hardware RAID 5 array, so hardware was definitely not an issue (unless the RAID 5 was tripping us up). I have heard the same complaints from other Beowulfers, so this seems to be indeed a major problem.

I would suggest you look to a Solaris server with maybe Linux clients, if you indeed want to deploy Linux (I am assuming, since this is /. after all). I haven't had any experience with such a setup, I just assume Solaris can do NFS the best :-)...

engineers never lie; we just approximate the truth.

NFSv3

[nconway]

The most obvious incremental improvement to make is to switch over as much stuff as possible to NFSv3 as possible (from what I've heard, you can get significantly better performance). I know that Linux 2.4.0-test support NFSv3 (as well as better overall NFS performance, even for v2), and there is probably a backport of v3 code to 2.2.x (if it's not already supported, I can't remember).

Mmmmm Origin!!

[Dr Caleb]

Don't make the same mistake I did with Origin systems!

If you are going to have a fairly disk-intensive app, don't use RAID 5!

It requires; 1 read per disk, 1 write per disk, then another read-verify from each disk. That's 6 I/O's per disk for a single write. If you need redundancy, use mirroring or a stripe set no parity (RAID 1 or RAID 4).

You'll get much better performance out of the disks!