NT faster than Linux in tests

Mike_Miller writes "The lastest Mindcraft Study claims that Microsoft Windows NT Server is 2.5 times faster than Linux as a file server and 3.7 times faster as a web server. Their white paper shows that NT beats Linux on every test. " Anyone have a critique?

Mindcraft's post to comp.infosystems.servers.unix

Fished out of dejanews -- the Mindcraft folks used the pseudonym '[e-mail]'

(If this was posted earlier, I didn't see it...)

Can anybody here respond to this?


Hi Everybody,

We're considering using Linux + Apache as a web server. The hardware
is a 4-processor 400 MHz (Xeon) server with 1GB of ram, a RAID controller,
and six disks. We have Redhat 5.2 installed and compiled an SMP version
of the 2.2.2 Linux kernel. For the web server we used the latest 2.0.3
version of Apache.

The scenario: we're bangin' on this web server with a bunch of clients
to try and get a handle on its capacity. Simple static HTML requests,
no heavy CGI yet. My Apache server is tuned up, MaxClients is 460.
I recompiled with HARD_SERVER_LIMIT set to 500. Limit on number of
processes is 512, imit on file descriptors is 1024.

The problem: the server performs well, delivering in excess of 1300
HTTP GET requests per second. But then performance drops WAAAY
off, like down to 70 connections per second. We're not swapping,
the network isn't saturated (4 x 100Mbit nets), disks are hardly used,
but the system is just crawling. If it were saturated then performance
should level off, not drop like this. Neither vmstat nor top show
anything unusual. No error messages in the web server. Its puzzling.

Any ideas? Any tips, suggestions, or pointers would be appreciated.
Thanks!

Erm..

Apache collapsed after 250 threads on a Quad 400Mhz Xeon? Something is definately screwy there..

I have average close to 60-70 Apache threads running as a regular load on Pentium-120 with 64megs of ram without any problems. Most of those are database-generated, rather than plain file GETs. Someone has been either drinking or got paid some dough..

Say what?

"The Linux kernal limited itself to 970 megs of RAM"... Say WHAT?

Really; the Winbox had most of its services shut off, while the Linbox was running SMB, NFS, etc. My guess is that they were probably hitting those other services while they were taking the numbers.

Besides, this runs contrary to every other (non-MS paid-for) study I've seen. Mayhaps someone should do some independent verification. Be sure check if the Windows numbers were a "demo".

Hey, they lied to Justice; why wouldn't they lie to us?

----

Will @ Whistlingfish

Thanks. I just checked that out. It does appear that they asked a single question about Apache performance. I remember seeing that posting myself and blowing it off because there wasn't enough info to tell him anything and I didn't feel like going into the give-and-take to get enough info to do something. (I do enough of that supporting my own customers!). Now, in hindsight, knowing what he did not do to Linux, the answer is obvious: he was running out of file handles. Do the math. An inactive Apache server has 8 file handles open. 127 servers max * 8 = 1016. Default file_max is 1024 for Linux, of which 150 or so are usually open while the system is at a rest. Apache could not bind a socket to a file handle for incoming connections because there were no file handles. So Apache was basically deadlocked, waiting for file handles to come free so it could accept() the socket but it was already holding all the file handles!

If there had been questions about general tuning of such a large system, that would have solved the problem because someone would have remembered about file_max. But one cryptic query that didn't give enough information to get help does not an honest effort make.

Anyhow: I guess I have to post a partial retraction. They did post a *SINGLE* query to the net.

-- Eric

Some notes on Mindcraft test ...

Apache 1.3.4 Configuration
Set OPTIM = "-04 -m486" before compiling
on 4 x 400 MHz Pentium II Xeon

Samba 2.0.1 Configuration
wide links = no
That creates a bottleneck in Samba performance, see here [linuxtoday.com]

the following processes were running ... (kswapd), /sbin/kerneld,syslogd,
not sure if that means something, but why they run kerneld with 2.2 kernel ?

On NT side:

Tcpip\Parameters\Tcpwindowsize = 65535
that makes huge boost on network performance, but only on local network where packets don't get lost

Set Logging - "Next Log Time Period" = "When file size reaches 100 MB"
Logs on the F: drive (RAID) along with the WebBench data files So basically server does much less logging than Apache - and since it's many small requests, and since Apache writes logs on a non-RAID disk all together it'll be a big bottleneck

Anyone noted anything else wrong with this benchmark ?
From all my experience it looks like pure crap

P.S. Why they needed NFS ? inetd ?

Some notes on Mindcraft test ...

-m486 is pretty standard. I would be very surprised to see a performance gain of more than about 1% between -m686 or similar and -m486.

wide links=no has been explained on other threads. It certainly slowed down performance by an unreal amount. This is a paranoia security measure that apparently some admins would use. I don't know that it's fair to assume that they put this in solely to skew the results.

As for kerneld, inetd, NFS, etc.: all right it's unnecessary, but will use under 1MB of RAM and under 0.1% CPU most likely. I don't see this as an issue.

My best guesses for the apalling results are something like this:
- the wide links=no thing. NT doesn't have to worry about symlinks. I think this is unnecessary in pretty darn well every case. This could either be intentional malicious intent for the pro-NT side, or inexperience/a mistake. Either way, it would be nice to show some number with this turned on.
- pure speculation here, but they may have set up Apache to do real-time hostname lookups. This is an absolute no-no for any serious server. Again, possibly inexperience or a mistake.
- the >512MB RAM Linux bug. I've heard horror stories, and I've read people with no problem at all. Also, I believe this was a problem with PIIs only, and they were using Xeons in this report. Who knows.

Anyway it appears to be a bad combination of very silly yet somewhat understandable (for newbies) software misconfigurations, and some bad choices in hardware. Which brings me to another point: quad Xeon for a server? My K6-166 could handle a few thousand hits a second I'm fairly certain. Adding more processors will only slow things down when you're dealing with file serving.

It's hard to say whether or not they did this intentionally. It's fairly obvious that they didn't know exactly what was going on with Linux. I'd say Microsoft has a list of hardware that they know works well, and when these people asked for sponsorship, some Microsoft people said "OK here's some hardware that we know works. BTW doing this and this and this might help out your performance". Not to say that Microsoft went out of its way to hurt Linux, but they probably know what works best on their own systems.

Erm..

More than something screwy. You can't use more than 100 threads on IIS 4. It uses Microsoft's Transaction Server thread resource pooling to do thread management, and MTS is internally limited to never handing out more than 100 threads. So basically at least some of it is incorrect.

Sponsors

Yeah, this study is sponsored by Microsoft, if you read the fine print:


Mindcraft Certification

Mindcraft, Inc. conducted the performance tests described in this report between March 10 and March 13, 1999. Microsoft Corporation sponsored the testing reported herein.


Looks like you can buy anything you want with enough money. It doesn't make it a true indication of a real-world situation.

I think that there's enough evidence to the contrary already out there, and this will only serve to discredit Mindcraft.

Also, it seems they crippled the Samba Server

According to a posting [linuxtoday.com] on Linux Today by Jeremy Alison of the Samba Team, it seems that the Mindcraft study crippled the Samba server in the tests:


From Andrew Trigell (original Author of Samba):

They set "widelinks = no" now I wonder why they did that :)

In case you haven't guessed, that will lower the performance enormously. It adds 3 chdir() calls and 3 getwd() calls to every filename lookup. That will especially hurt on a SMP system.

We should learn from this

Okay, folks. So we have a bit of egg on our face for this one, because nobody (to my knowledge) has really stepped forward with large-server Linux benchmarks which demonstrate anything differently. It may be that Mindcraft royally screwed up, or it might be that Linux really is slower than NT for a certain set of benchmarks -- the truth is more likely a combination of these factors.

If Linux is going to be treated as a serious operating system by the majority of the IT community, it's going to have to step up to the plate and demonstrate scalability and performance which does rival NT server in this area. Most of our knowledge about Linux-vs-NT performance is somewhat anecdotal -- we haven't really "put our money where our mouth is" and shown objectively that Linux can outperform NT in these areas.

Rather than dismissing this study as FUD, I think we could learn a few valuable lessons from this study. We should seek to understand why the benchmark results weren't as great as we would have liked. We should fix any obvious bugs or misfeatures in Samba, Apache, and the Linux kernel which stood in the way of higher performance. And we should stive to improve the entire system to make it be a true NT rival.

We have a lot going for us. First of all, we can innovate at a much more rapid pace than Microsoft -- so hopefully within just a few short months (and I'm being pessimistic!) we could demonstrate a high-performance Linux file and Web server which kicks NT's butt all over the place.

Nobody said building a high-performance, scalable Internet server operating system was easy. Let's get to it!

Matt Welsh, [e-mail]

Articles from our dear friends at Mindcraft.

An AC posted:

The net posts asking for help that are mentioned in the white paper appear to have been most likely made under the pseudonym:

[e-mail]

Use DejaNews.

No-one seems to have done that and talked about it. I did; h ere's the relevant link [dejanews.com] that lists all the messages from this guy on Usenet. Take a look at them and post what you think about them. It seems to me he hit a strange, obscure bug in GNU, Linux, or Apache, and it might have something to do with network adapter or SCSI adapter problems.

4 Network cards...

So they have 4 network cards in that nice little Dell machine, and they specifically mention "Used the affinity tool to bind one NIC to each CPU", so my question is whether they even bothered to use the other 3 NICs under Linux.

It seems to me that Linux with one network card doing only 2.5 times under NT with 4 network cards sounds about right. Give Linux 4 network cards and you get performace that easily blows NT away.

To use multiply NICs you have to use the network driver as a module I believe, did they bother to do that? I can't imagine Red Hat's installer "How many network cards do you have, but then again maybe it does, I'm a Slackware kinda person...

strange results / performance issues

The beowulf newsgroup had a couple short threads a couple months ago about consistently abyssmal performance on redhat 5.2 SMP machines running 2.x with > 512 MB of RAM. The two threads [ one [beowulf.org], two [beowulf.org]] deal with users who had horrendous performance problems with their new machines (both running 2.2.2, the same kernel as in the report) when they used more than 512 MB of ram, but the performance jumped right back up when they used 512 or less. Check out the articles to see how bad the performance was; it's pretty surprising, and presents an interesting opportunity for detractors of linux:

Linux definitely has some hardware/kernel combinations that would seem OK by design on paper, but exhibit peculiar behavior in practice, especially with SMP. I wouldn't rule out the possibility of the testers (or financial backers) hand-picking kernels/hardware configurations that could affect results while seeming perfectly viable to the layman.

It seems very likely to me that if Microsoft did not outwardly donate the hardware to the testing company, they at least made suggestions on its configuration. The open nature of linux development and bug disclosure could easily be used by companies wishing to stage biased demonstrations; Microsoft almost certainly does a thorough job tracking linux kernel development and bug reports.

-- Scott

Look at the OS configurations

Look at the OS Configurations:

For one, NT used 1Gb of ram will Linux used only 960MB. Surely they could have passed the parameter MEM=1024M to the kernel ...

Additionally they tuned tcpwindowsize under NT to 65536, and adjusted buffers on the network card to 200 (from 32).

They made no TCP/IP stack adjustments OR adjustments to the netcards under linux.

Just look at the sections explaining the myriad of things they did to "tune" NT. Then look at linux. Enable NFS. The following daemons were run. blah blah. Didn't bother to work on anything.

James

IIS has the real world proof anyway...

The fact that Microsoft won't use their own servers on one of their own sites, shows how much they rely on their product. If an NT server isn't good enough to handle their own web services, why should it be good enough to handle mine?

Response at lwn.net

Please submit any inconsistancies you see in this document (and if you don't see any, please shoot yourself in the head) to [e-mail]. They are readying a response as we speak.