Virtualizing Cuts Web App Performance 43%

czei writes "This just-released research report, Load Testing a Virtual Web Application, looks at the effects of virtualization on a typical ASP Web application, using VMWare on Linux to host a Windows OS and IIS web server. While virtualizing the server made it easier to manage, the number of users the virtualized Web app could handle dropped by 43%. The article also shows interesting graphs of how hyper-threading affected the performance of IIS." The report urges readers to take this research as a data point. No optimization was done on host or guest OS parameters.

Virtualize this

That is all very well, but we all KNOW apps slow down when we run them in a VM. What difference does it make to the average n00b who wants to watch funny videos [digg.com] and check their email? Anyone using computers for serious numbercrunching obviously won't virtualize anyway. No big deal

Re:Virtualize this

I do like the idea of a variably sized beowulf cluster running a floating number of package (LAMP) servers. Get more clients? Add more VLAMPs. Things slowing down? Add more hardware.

You still take performance hits, but if you can scale your system by just adding cheap commodity systems, that works. Plug it in, boot it off a CD, and let the Cluster take control.

holy cow am I a nerd

I do like the idea of a variably sized beowulf cluster running a floating number of package (LAMP) servers. Get more clients? Add more VLAMPs. Things slowing down? Add more hardware.

I started getting aroused as I read your post. This is highly disturbing.

Well,

Duh.

Seriously. I don't know who gave anyone the impression that virtualization was a performance booster. Management improver? Sure. Stability insurance? Why not? But if you don't get that virtualizing your servers imposes a bit of overhead, then you're probably not paying attention.

I especially love the idea that running different types of server virtualized on the same machine is a good idea; the idea of virtualization of multiple servers is to distribute the load. If you have, say, ftpd, httpd and mysqld running as their own virtualized systems, they will all get hit *simultaneously*.

Again. Duh.

Re:Well,

Seriously. I don't know who gave anyone the impression that virtualization was a performance booster. Management improver? Sure. Stability insurance? Why not? But if you don't get that virtualizing your servers imposes a bit of overhead, then you're probably not paying attention.

Well, I think the point was that he attached an actual number to the amount of the performance hit, which is relevant. That's called research; quantifying and proving that which seems 'obvious'.

Re:Well,

Well, it's not a surprise, but it's probably worth quantifying.

Here's a question: what is more available: hardware or skilled system administrators? Obviously hardware.

Here's a common scenario: you've set up a system to provide some useful package of services. How do you let other people duplicate your success? (1) tell them what hardware they need and (2) have them install and configurethe software on their hardware. Guess which item involves the most cost in the long run?

The hardware is easy; the greatest barrier and cost is the process of installing and configuring the software. That's one place a virtual machine is worth considering in producation systems. You aren't going to use something like VMWare in one-of-a-kind production systems. You're going to use it when you need to clone the same set up over and over again. This is very attractive for application vendors, who spend huge amounts of support on installation and tracking down compatibilty conflicts.

Another application would be an IT department that has to support dozens of more or less identical servers, especially if they are frequently called upon to set up new servers. If I had a choice, I'd use Linux virtualization on a midrange or mainframe, but if those servers must be Windows servers, then I'd be looking at some kind of cluster with SAN. This is not really my area of expertise, but we're talking high end boxen for x86; if the typical server didn't need 100% of box, then I have three choices: waste server bandwidth (expensive), force groups to share servers (awkward and inflexible; what if I have to separate two groups joined at the hip?), and virtualization.

Naturally if you are virtualizing production servers, you need to scale your hardware recommendation up to account for VM overhead.

What would be very interesting is a study of the bottlenecks. If you are considering a system with certain characteristics (processor/processors, memory, storage/raid etc) and you have X dollars, where is that best spent?

Re:Well,

To us, the whole point of virtualization is that we have several servers which are mostly idle at all times and completely idle at most times, and for support reasons we're not supposed to be running anything else on the same copy of windows. So we can replace five or six systems with a four-core 1U box with just a few gigabytes of memory, which will recover rack space and reduce power consumption. For anything that's actually heavily loaded, running on the hardware is probably a very good idea.

Bogus Test

Who uses VMWare Server in a production environment anyway? We run all of our Web services, Exchange servers and SQL databases in VMWare's Virtual Infrastructure 3. VMWare Player and Server are only ment for lab evironments and low load applications. VMWare even says as much on their website. Either this is just FUD or the author is an idiot. In other news water is wet.

Re:Bogus Test

Yes: it performs much, much better.

VI3 is actually a suite of products. At the heart is VMware ESX Server [vmware.com], which is actually an operating system in its own right: it runs "on the metal," without having Windows or Linux installed already on the system. It also has a service console operating system which looks suspiciously like a *NIX style operating system, so you can SSH directly to the system, cd into your /vmfs directory and, say, scp disk files over the network. If you wanted to.

However, as a pretty damn safe rule of thumb, no system is going to run faster on equivalent hardware after being virtualized. In a prior job where I was often asked to provide development/test systems, I got phone calls from a lot of people who were bitten hard by the virtualization bug. Whenever someone brought up any issue having to do with infrastructure, no matter how odd or off the wall, they wanted to push virtualization as a solution. I had to explain to them that if your problem is that a web server is slow, the answer isn't to install VMWare server on it, set up two host operating systems, and say, "There! Now I have two web servers." You'd be surprised how pervasive that sort of thinking is, even among people who should patently know better.

Another useful guideline: various types of services are impacted differently by being virtualized. Generally, the best candidates for virtualization are ones that spend a lot of time idle. This is actually more common than you might think - people need a server set up for something, can't put it on a pre-existing system for security/compatibility reasons, so they go out and buy a new system which is ten times more powerful than they need. You can put a lot of these kinds of systems on a single, reasonably powerful ESX server. On the other hand, systems that heavily tax available hardware, especially I/O, are usually much harder to deal with.

Re:Bogus Test

Actually, as it's been explained to me, the ESX hypervisor itself is pure proprietary code (and small, too). The Service Console is very readily admitted to be a tweaked out RHEL (3, I believe..). Linux is used to boot, and then (magically?) transfers control of the bare metal to the hypervisor. Linux then jumps into a virtual machine, although it's not presented like a virtual machine, which creates all this confusion.

In the end, the tweaked RHEL that you interact with (ssh, scp) is not the hypervisor, but a VM with special tools that can manipulate the hypervisor.

Re:Bogus Test

Actually, the company I worked for 6 months back, one of the projects I was involved in was 'VMWare'. Production stuff running on on the ESX servers (which became 'virtual infrastructure') in our datacentre, as a cost effective scalable environment. Yes, we weren't getting 'uber performance' but then again, we were running 150 or so VMs on an 6 server VMWare farm.

One of the other things we prototyped and deployed was 'site services packages' - get GSX (now VMWare Server), stick it on a pair of 2U servers, and attach a storage array to both of them. Then create your 'template' fileserver, DHCP server, print server, proxy, that kind of thing and deploy them to this package. It worked very well indeed - you get a whole new order of magnitude on stability (although to be fair that's in part because we through away the crappy workstations that were doing the 'low intensity' stuff) and was extremely managable, and trivially replacable in the event of a hardware failure.

Performance? No, VMWare isn't that great on performance - whilst it's not bad, in an ideal situation, fundamentally what you are doing is introducing an overhead on your system. And probably contention too. But it's really good at efficient resource utilisation, easy manageability and maintainability.

As an experienced sysadmin, my reaction is screw performance. Let's start with reliable and scalable, and then performance just naturally follows, as does a really high grade service.

Proactive laziness is a fundamental of systems admin. Your job, is essentially to put yourself out of a job - or more specificially, free up your time to play with toys. The best way to do this is build something stable, well documented and easily maintainable. Then your day consists of interesting stuff, punctuated by the odd RTFM when something doesn't work quite right.

This is VMware Server and not ESX Server

They performed the test on VMware Server not VMware ESX Server which is what most enterprises will use. VMware ESX Server runs on "bare metal", so it does not have the overhead of the host operating system.

Re:This is VMware Server and not ESX Server

I think everyone is kind of looking at this the wrong way. Sure you get a performance hit, but yu are testing maximum performance. That is a situation where you wouldn't want to virtualize anyway. If the system is running at 100% utilization, then leave it alone. It is more interesting to take your servers running at 20-30% util (if that, how many idle server do YOU have?), and cram them all into a couple of boxes. You most likely WON'T see a perfromance drop because there was so much headroom on the system already. Virtualization struggles at the max utilization case, but then that is not the case that it is really meant for.

This has been my experience too

Linux under VMWare's network performance is pretty bad. An interesting visual confirmation is to use an ssh shell and watch the lag. That may just be the broadcom chips in the servers the company I was working for used, though. Guest OSes are fine for some low traffic stuff that only a few people will be using, and is definitely the way to go in the test lab; but I wouldn't use this configuration as a company's primary reverse proxy or mail solution.

That said,
I use a windows vmware session under linux for those times I have no choice, and it works just fine network-wise as a workstation.

Sounds about right

My first attempt at virtualization was last September with VMWare Server. During testing everything seemed fine. When everything was using it, performance was awful. Everything crawled. I ended up doing an all-nighter to move everything back to a regular server. Note, I wasn't overloading things. There was only one VM on the host. The memory was fixed, not paged to a disk like it is by default. The hard drive was preallocated. My intention for virtualization was to make things easier to manage.

That's when I started experimenting with Xen. This time I put the test under a very high load, and it seemed to handle everything well. I deployed it in October and so far there hasn't been a single performance issue.

I'm now totally addicted to Xen. I create Vms all the time, have split up services into different VMs (ie, when cups crashes it no longer takes out the copy of samba that handles logins, damn I hate cups). So far, no performance issues at all.

single data point is correct

Dell Poweredge SC1420 with dual Xeon 2.8GHz processors

While I can't seem to find all the information on the SC1420, it appears as though this product uses processors from the Prescott generation of Intel CPUs. Some chips from this group support "Vanderpool", Intel's hardware virtualization solution, but not all do. The presence or absence of this feature could greatly impact the performance penalty faced by operating a virtualized computing environment. Further, Intel's new Core2 based CPUs feature a hardware virtualization implementation which may have vastly different performance characteristics. AMD's K8 family supports hardware virtualization as well. I'm excited about their new line of CPUs based on the K10 (Barcelona) core, which feature "NestedPageTables," which are supposed to greatly reduce overhead by doing memory translations in hardware instead of in software by the hypervisor.

All I'm really trying to say is that this article really is only a single data point. I wouldn't let their results influence your overall view of virtualization in any way...

Re:single data point is correct

The biggest overhead from most forms of virtualisation is from emulated devices. If you have loads of money, you can give it to IBM and get some hardware with virtualisation-aware network and block device controllers. Then you get good performance. Alternatively, you can use paravirtualised device drivers. Xen supports this by default, and I think KVM does now for networks. Not sure about VMWare.

With paravirtualised devices, or devices that are virtualisation-aware, a VM can be within 10% of the performance of a real machine quite easily. Without I'm surprised they even got to 57% of native performance for web applications.

Pointless test?

Come on! You run virtualised web servers because 99.9% of all web servers are idle at any given time. So you put 100 on a server. The customer doesn't see any worse performance with their 3 hits a week page and the ISP makes more money/server.

Re:Pointless test?

AssignUserId only works with the perchild MPM, which has the following caveat: "This module is not functional. Development of this module is not complete and is not currently active. Do not use perchild unless you are a programmer willing to help fix it."

Thus, AssignUserId should NOT be used. SuExec can be used, of course, but that has its own limitations.

Personally, I give users their own Apache processes on their own port (>1024) and use a reverse proxy. I make a living on it.

Re:Pointless test?

No it's not insane. Lots of customers want full root access on their systems so they can install whatever they want (different database or other servers, or even alternate OS's). Virtualization is the only way to go for that.

Re:Pointless test?

It's not insane if people want different solutions or even want their own server. With virtualisation, a host can offer multiple php versions. You can avoid all the security problems where one script running as the webserver can read any other file accessible to the web server.

You can also get better management control of resources, preventing one site from eating up all available resources on the box.

That's not to say there aren't a million good reasons to use virtual servers in apache, just to point out that virtualising web hosts is not, by definition, a daft idea.

Re:Pointless test?

You run virtualised web servers because 99.9% of all web servers are idle at any given time. So you put 100 on a server.

If you have a real need to run 100 separate Apache instances, then you'll want something much higher-level than VMWare. For us, that would be a FreeBSD jail, where each instance would get its own chrooted home directory and IP address. That way, you're not allocating resources to 100 little-used OS images; each shares from the same memory and hard drive pool. Jails are slightly limited in that I'd like a way to limit CPU and memory allocation, but in practical application this really works very well today.

Hidden advertisment

This smells like a hidden advertisment for "Web Performance Inc.". Now somebody please tell me why I should trust the results produced by a relatively unknown product and company, and not stick to proven tools like Borland SilkPerformer [borland.com] or Mercury Loadrunner [mercury.com].

Bad data, bad setup

There's quite a lot wrong with their setup.

1) As others have pointed out, they should be running on ESX to get best performance.
2) Physical machine was a dual-proc. How many processors did they assign to the VM?
3) Physical machine had 2GB memory. They assigned 2GB to the VM!! Vmware will take 256MB of this
for itself, so that 2GB visible to Windows will be being swapped.
4) How many disks did the physical machine have, and what was on them?
If e.g. the physical machine had two disks, the VM should have been given two disk files, with each file being placed on a different physical spindle.

You get the picture.

well duh

Do i need to say more? We all know running any VM app is always going to be slower than a real OS on real Hardware. Ok mabe I didnt realise how much slowdown their was but there are easier and cheaper ways to run web servers. Nice to have given it a go and gathered stats but nothing new.

data, datum, data

I know that the term 'data' is generally used as a singular in informal speech, but it still drives me nuts. I bet there are a lot of other current or former Latin club members that howl at this literary fingernails-on-chalkboard usage of the term. Not that I stayed after school for Latin club meetings, or to play Civ, or D&D...

Silently weeps into his tattered copy of Remedia Amoris

-Foo

"Duh!" moment

I agree with many of the commentators here that this is pretty obvious. We use virtualization a lot, but also realize its limitations. For example, we don't run SQL or anything heavily transaction or I/O bound. CPU utilization is usually not a problem; virtual machines perform as well as their physical counterparts in most instances unless you have a lot of CPU intensive virtual machines running.

Web servers are mostly memory and CPU bound which would give one the impression that they would be great candidates for virtualization. However, VMWare Server is not the solution; network I/O is not good on Server. Typically your results would be maybe 75% of the actual physical speed on a "passthrough", less on a NAT. It depends a lot on how your network is set up, not to mention the abilities of the physical machine.

The best solution is Virtual Infrastructure (used to be ESX). That product tackles most of the failings of VMWare server and fixes them. The only exception is that I still wouldn't run anything I/O heavy on VI. SQL's a no-no. Also, if you're not getting the performance from a single web server that you expect, you can easily throw up more web servers. Now, obviously you might get into M$ licensing issues, but that's why you run your web services on Apache :D

Use Virtuozzo

Of course VMware and Xen are going to be slow - that is the tradeoff you get when you want the ability to run both Windows and Linux at the same time.

http://openvz.org/ [openvz.org] - it does a much better job of virtualizing IMO. The only minus is that all VM's have to use the same kernel version.

How does the webapp scale?

Unless it scales linearly with the number of users, that is a pretty useless metric for the performance of the virtualization system (No, I didn't RTFA).

Obviated Comment

I have decided to refrain from commenting on this article, as I do not wish to be tarred with the epithet "me-too-er".

Having said that, "Me, Too!"

Fast Virtualization: Xen, KVM, Virtuozzo, GSX, ESX

For speed, the newer virtualization tools KVM, Xen, and Virtuozzo are presently substantially ahead of the present incarnation of VMWare. KVM requires the new "hardware virtualization" CPU's from Intel and AMD which must be mentioned here, since they represent a major industry recognition of the value of virtualization. This article seems to be giving people the impression that performance of VMWare Server is indicative of virtualization tools in general, and that all virtualization tools slow down hosted virtual machines dramatically. This is simply false. I know hosting providers running 50 virtual servers on a single dual CPU box with thousands and thousands of users, which would simply not work if all virtualization tools had a 43% hit per instance. Another key matter here is that the author fails to mention (or realize?) VMWare Server is crippleware. VMWare states explicitly not to use VMWare Server for anything other than testing because it does not have the performance or feature set of their full blown ESX and GSX servers. Also, while VMWare may be the oldest and arguably most mature virtualization suite, it is certainly not the fastest.

People still use IIS?

who are these people? why are they not being publicly flogged?

Quantifying

I think it was pretty obvious that it would add overhead and therefore drop performance. This study is good for quantifying how much of a performance threat it is. And 43% is not an incredibly bad value in my opinion, it is not even 50% ...

These results are pretty much as expected

It isn't surprising that VMWare would be bad at a web-app workload. See the original paper on Xen:

http://www.cl.cam.ac.uk/research/srg/netos/papers/ 2003-xensosp.pdf [cam.ac.uk]

Top of page 9 has a chart comparing native Linux, Xen, VMWare, and UML for different workloads. They show VMWare degrading performance by over 70% for SPECWEB 99.

Web applications are OS intensive; while VMWare is quite good at pure CPU-bound tasks, it has to perform a lot of emulation whenever you are running inside the OS. So it will stink at anything with lots of small IO, lots of metadata operations, or lots of process creation/switching. For example, VMWare shows a whopping 90% slowdown for OLTP database workloads, according to the Xen paper, and it really isn't surprising. The OS microbenchmarks in the above paper (page 10) show that VMWare has abysmal performance for things like fork(), exec(), mmap(), page faults, and context switches.

Basically, Xen doesn't have to emulate the OS, because they make modifications to the OS. VMWare does dynamic binary rewriting (think fancy emulation) to run an unmodified OS; they therefore pay through the nose in performance overhead for OS-intensive workloads.

"Typical Application" they for sure ment 42%

What what the typical application. Was it well writen or not. what was the reason for the slowdown (Memory, Network, ?). Without that 43% is just a munber as good as 42%, the answer to all performance loss questions

Xen Scales

The CPU-overhead on the Xen hypervisor is much much lower - between 2-4%. http IO has been less explored. There have been workshops on its use in HPC http://xhpc.wu-wien.ac.at/ [wu-wien.ac.at] http://xhpc.ai.wu-wien.ac.at/ [wu-wien.ac.at]

That's easy to fix!

Since a virtual server only gets 57% of the performance of a physical server, just run TWO virtual servers!

That way, you'll get 114% throughput!

Next week, I solve world hunger, global warming, and bring peace to the Middle East.

Would be interesting if done right

1. They fucked up their setup. They assigned 2Gigs to VM and all host has is 2Gigs too? Brilliant.
2. Since when are you allowed to post benchmarks of MS software?

VM performance comparison

Has anyone done a recent true apples to apples comparison with Vmware, Virtual Server, and maybe Sen on the same exact hardware running the same guest VMs? Obviously it would have to have Windows as the host OS due to Virtual Server being Windows only but I would like to see how the various solutions stack up in different scenarios. Such as IO heavy, CPU heavy, sheer number of VMs that can be hosted, etc.
---
You can use any kind of HTML formatting that Slashdot accepts.
Generated by SlashdotRndSig [snop.com] via GreaseMonkey [mozdev.org]

really bad report...

But it highlights one thing: if you hand virtualization to clueless people, you'll get bad perfs.

It also shows, both in the article and in the comments here, the severe misunderstanding surrounding the concept of "virtualization".

I see lots of clueless people saying "uh, of course, virtualization perfs sucks". I think those people don't realize today's virtualization technology ain't grandpa's past-century emulators.

There are today virtualization technologies that offer basically native speeds. Xen can now run in two modes (para-virt or hardware-virt, the latter if the MOBO/BIOS/CPU supports Intel-VT / AMD-V)... In paravirt mode Xen offers native speeds (the overhead is so small you'll have a hard-time measuring it). Better: network I/O ain't good enough for you? Simply "passthrough" a PCI device (say a PCI network card) to your paravirtualized guest. The guest (and only the guest) is directly accessing the PCI card (no more network I/O problems). But you can't run Windows on Linux using paravirt under Xen...

In hardware-virtualized mode, under Xen (or KVM, which only does hardware-virt), you can run Windows. Network and disk I/O, for hardware virt, at this point sucks. However you can install special drivers in your guest to make it speedier (drivers for Windows under Xen are $$$ and under development for KVM).

But, wait, there's more to come... Next gen IOMMU is around the corner. And as soon as it gets implemented in Xen, the already super-fast virtualized system gets an additional boost and you'll have something even closer to native, even when running Windows under Xen.

If you think "virtualization will always be slower" you need a reality check: the CPU makers are working hard so that the virtualization overhead becomes irrelevant. And suddenly the ones not using virtualization will find themselves with a less capable, less secure, less maintanable box being, in some particular, anecdotical, cases only 0.05% faster.

Virtualization is here to stay and the overhead, already very small today, will keep shrinking.

VMWare EULA?

Although the focus is on virtualization and its effect on performance, I noticed that the author disclosed that he ran the tests on VMWare. The VMWare EULA [vmware.com] states in section 3.3 "You may use the Software to conduct internal performance testing and benchmarking studies, the results of which you (and not unauthorized third parties) may publish or publicly disseminate; provided that VMware has reviewed and approved of the methodology, assumptions and other parameters of the study.".

This relates to the earlier slashdot story where xensource [xensource.com] had to redact areas of their performance whitepaper [xensource.com]

I wonder if they gave him the green light to report these results on VMWare?

Seriously

This is stupid!

First of all, VMware Server was never intended to be used as a dedicated server, even when put over Linux! That's plain dumb! VMware made it available so that customers can demo the product before buying it.

If anyone wanted to run a dedicated virtual server, then they should use VMware ESX Server.
To obtain semi-realistic statistics on how virtualization is improving, then such statistics should be taken from test on ESX, an environment set up for actual server use; If possible, test the VM on different versions of ESX.

And how is that a load balancing test?? Using one machine only!?? A load balancing occurs when you have multiple machines and distribute the workload on all of them.

Silly test.

Apples to oranges

Virtualization will slow things down... yes that's obvious, since you're emulating the hardware to a certain extent, and triggering a bazillion context switches every time I/O happens. This stuff is bad for web apps, sure, but normal people don't run just one VM per box. The benefit comes from having multiple VM's running on the same physical machine. For many outfits, it's far cheaper and more efficient to have a whole bunch of little isolated VM's running on one big beefy SMP server, than to have them all as little boxen taking up tons of space and chugging more electricity than Tokyo Square on payday. It also makes it a lot easier to migrate those VM's to bigger/better gear as needed for upgrades or failover protection, often with zero downtime (just a bit of lag during transfer). That's how you can justify spending 25k on a 16-core SMP beast with 32gb of ram and four independent RAID-5 banks. It's still cheaper than 16 separates, not just the upfront cost of equipment but also the ongoing maintenance costs.

Hell I even have a VM server in my apartment, just for the convenience of running 4 distinct OS'es on one neatly stashed box. I don't need a KVM switch or long-ass serial cable, everything's done VNC-style through the VMware console from boot to poweroff. That feature alone makes VM's immensely attractive in many scenarios. For sysadmins, you can do all the work from your desk, no need to go stand in the chilly server room for an hour waiting for a tape restore or OS reload, and you can even script a lot of functionality through the hypervisor API. I'm not saying it's the solution to everyone's problems, but for those things it was designed for, it does them extremely well.

Good News!

A study shows that even without optimization, web apps can ran at almost 60% of their native speed using virtualization. Using better optimization, this percentage can be even higher. It's all how you present the data!

Hyperthreading

I thought the "Does hyperthreading help in the realy world" issue was related to the way Windows does task switching.

I am under the impression that hyperthreading helps Windows more than Linux because Windows fails to save certain register states, and thus incures a higher cost, in terms of performance, when it task switches.

His test uses Windows VM's on top of Linux. Thus I could see when it could help in his situation, but believe that the generalization about hyperthreading is misleading.

Can anyone clarify this?

Virtualization comparisons

This is obviously one data point, and as others have mentioned, not even the best point for deployment.

But there's a whole raft of virtualization solutions available, and that's just in the Linux kernel, not to mention the Windows solutions. It would be fun/interesting to see an updated comparison of the various solutions.

Then for the real benchmark point, it would be good to see what IBM does with the Big Iron virtualization. Intel and AMD are finally adding hardware support, and it sounds like Intel is improving its support on upcoming products, and no doubt AMD will be, as well. But IBM has been virtualizing for decades, and IMHO has probably forgotten more than Intel and AMD have ever learned.

VMWare, not Virtualization

VMWare uses one kind of virtualization, where it intercepts system calls (by loading a module into the kernel that traps faults to privileged instruction execution) and emulates them. When doing heavy amounts of system calls, like using TCP/IP sockets or opening files or reading files that are already open or writing to the UNIX socket for a database (which reads and writes to disk a ton) or whatever, the system slows down massively. Paravirtualizing Hypervisors like Xen rely on the guest OS having its code rewritten to specifically call the hypervisor, in which case the overhead of a syscall is... a syscall. For the most part, performance hit is negligible.

Ancient Hardware

Hyperthreading? Good lord, this is running on ancient hardware. This should be deployed on something that can use either AMD-V or Intel VT. VMWare has hyperthreading support and can show some improvement with "snappiness" but it doesn't seem to help in general throughput over the long run and may contribute to instability.

Usually, the limiter in this type of setup would be IO. When one virtualizes such a setup, you must reconfigure your application to minimize disk IO (with web servers we cache like crazy and really jack up the ram. 16-64 GB is my norm if it is a highly transactional site). 2GB is definitely not enough for IIS & Server 2003 doing anything more than serving static pages.

ASP ASP.NET

The headline says the application uses ASP. The article states they used ASP.NET in their tests. This is a large performance difference, primarily because .NET is byte-compiled at execution. ASP is purely interpreted.

This is comparing single to dual proc

They state in the test that the servers are dual proc servers.

VMWare Server, the free edition only emulates a single processor environment for your virtualized host.

VMWare ESX or whatever they are calling the expensive thing today, has the ability to give your virtualized host multiple processors.

So it's not surprising that it could only handle half the load, it only had half the processors.

We don't do virtualization for heavy use environments. We do it because different business groups don't want to share servers... that is, they can't agree on maintenance windows, etc.

no brainer. Don't virtualize stuff with high I/O!

Hmmm virualization 101. Don't virulize anything that requires high I/O if you do not have the hardware to do it.
First of all I built an ESX server farm for high I/O apps. Feel free to search for my name at vmware. I used to work
at Welch Foods until the new CIO just mentioned to words "Outsource IT" I left before a decision was made so I can
"stick it to the man". What the heck, the IT Director left anyway. Anyway we created an esx server farm with 8 Dell
6650s, and we had 2 Dell 6680 prototypes to evaluate. Most of these machines had 48GB of RAM and 4 GBIC connections
to our EMC SAN. And... Lots and lots of disk thrown at the databases!!!!!!!!

Another prime candidate to use for a server farm would be a BLADE server with 2 dedicated GBICS per blade.

If you decide to put up esx server on one server with just 2 scsi controllers and expect to run multiple virual machines
to support multiple web applications, then you are a dumb ass and should get out of IT. :)

If your a company that would like an ESX 3.0 server farm designed, then send me an email

My next consulting gig will be Delta Dental.. A very fast esx server farm ro run Oracle on Linux. .. and yeah it can
be done. You need the right hardware and software know how. TUNING.. lots of tuning.

half-assed install = half-assed performance

If I read this right, it's an unoptimized install on a barely server-grade system using a dev package rather than a production package. I would *expect* it to run less than stellarly.

This translates to "brother-in-law IT is crap."

heck, it might be better to say "VMware server will limit performance degradation to 43% when used in a poorly thought out implementation."

CPU's Get Faster

As long as CPU's continue to get faster and cheaper, and caches larger, this is hardly a gloom & doom end of the world. Most systems don't tax their CPU's most of the time anyway, and for the next 3 years at least CPU's will continue to get faster and cheaper.

VMWare w/ Windows VM?

Um duh! I thought the idea behind virtualization was to run what would have taken two computers on the same computer, and simplify administration. Who really has a mission critical high volume server running in a VM machine? Shakes head... I've used it to take four old (old) servers and put it on one new computer which is actually an upgrade from the P233mhz machines they were running on. I can also see user environments using VMWare to make it easier to restore computers when ID10T's screw something up. But never for a high volume web server. Besides IIS calls way more threads than Apache does to host a page, why would you use IIS for a high volume web server? I do appreciate the study pointing out the obvious!

The perfect VM ecosystem

In a perfect world, all VM images would use a common file format and a common subset of emulated devices supplied by all VM engines, e.g. ABC network card, DEF printer, GHI sound card, JKL video card, etc.

Also, the OS vendors like Microsoft and Linux distro vendors should get together with VM vendors and make their installers install the OS "vm-ready." This way, if a customer tried to boot his server's hard drive in a VM environment, it would come up, see the hardware change, and continue along its merry way. Likewise, the OS vendors should create tools to make it easy to move a VM drive to actual hardware. This may mean making an "interrogation CD" that can boot a new box to create a list of hardware, which can then be used to install the correct drivers on an existing VM'd installation in preparation for migration. I realize some of these tools already exist but the industry needs to standardize.

VM engines would distinguish themselves by their relative efficiencies, costs, management tools, additional virtual devices, what hardware devices they pass through to emulated VM devices, what devices were passed through "direct" as virtual rather than emulated devices, etc.

Also, ideally, there would be a popular, easy-to-set up, minimal-overhead, open-source, reasonably-efficient, reasonably-easy-to-manage VM designed for cases where 1 VM ran on each real box. In the simplest case, this would amount to nothing more than

• a minimalist operating system, say, a stripped Linux distro
  
• a minimalist management interface, just enough to create and delete VMs and start and stop VMs, running only 1 VM at a time
  
• the VM engine itself
  
• VM engine is agnostic to guest OS, it doesn't care if guest is Linux, Unix, Windows, or other.
  

Over time, performance is better

Because eventually, over time, a disaster will occur causing 100% downtime (or 0% throughput) on a monolithic system. Being able to restart the virtual machine quickly or even transparently on another piece of hardware without doing restores, hardware driver installations, or any number of other time delaying obstacles will increase the overall performance and total throughput of the application. In other words: A monolithic system can calculate pi to 10,000 places faster than a Virtual Machine. But they might both get to 10^10000000000 places in similar timeframes .

Of course it's that way

Te purpose of VM is to consolidate servers. The reason yu'd want to consolidate is that you find you are running 4 physical boxes and each one is running at about 1/10 of full capacity. So you say "cool I can loose three boxes and cut managmant and electrical power by 3/4. On the otyher hand if you find you need more performance you buy more boxes and "load balance" them

Also everyone knows you loose some performance running VM but if you need to run OS "A" guest or OS "B" host this is the only way. If you need to run Solaris on Solaris there are faster and better ways (zones) and same with Linux (zen)

The whole point of VM in a data center is to put "extra" cycles to better use, you would never use if if "cycles" were in short supply.

In other news...

Webperformanceinc later released the following reports:

• A faster connection to the Internet may lead to better response time from Internet servers.
• Static pages can lead to better server performance than dynamically generated content
• Powering down servers can save electricity, but leads to up to 100% decreased performance
• Michael Jackson may have taken plastic surgery

Basic computing

This is pretty basic stuff, I think everyone knows that when you use a VM it won't run as fast as hardware. This is a classic tradeoff of management vs. speed. What is more important? Choose your solution from there. Although I think a good way to setup your servers is to use a hardware server and have a VM box for cutting over in case of failure on the main server. This way you at least have something running and aren't SOL if your server dies.

Slashvertisment

The test was so poorly performed that the article was useless. It is little more than a poorly disguised ad for their load testing product. All I want to know is: how much would it cost for me to post a similar slashvertisment as a'story'?

KO (coke) used VMs for threading problem

In the late 90's I heard about CocaCola in Atlanta using virtual machines to increase the number of users per Lotus Notes server. When running Lotus Notes servers on Windows they found that they ran out of threads even though the procesor and disk load were not maxed. By running linux on the HW and running multiple Windows VM's for Lotus Notes server they were able to have more users per physical box.

At the time they were running out of server space and power in the server rooms so there was a rule that one server had to be sunset before another was added. By adding multiple virutal machines per server they were able to increase the number of users w/o adding more physical servers.

SQL Server was virtualized too

I have exchanged a couple of emails with Chris Merrill (the person who did the test). He confirmed that SQL Server was used as the backend for the ASP.NET Issue Tracker System (the app they were testing) and that SQL Server was virtualized too (I am not sure which version and which edition of SQL Server they were using). My experience have been that database servers do not virtualize well, at least not the ones under any kind of decent load. Unless we are talking really small or very mildly used web apps, you do not normally put web tier and database tier on the same box (at least not in the Microsoft world). It would be very interesting to see the results with SQL Server removed from the equation (running on another server).

VMWare Server instead of ESX?

why in the bloody hell were they testing with VMWare Server 1? If they want to test a production business app then they should have used a tuned VMWare ESX machine and tuned guest OS. As far as I'm concerned VMWare Server, while a good product (and better than MS's server product) is not something to be used in production, it's more for labs/dev environments, or very small companies that have no budget for virtualization and can live with the performance decrease.

I also see they used cheap hardware, something meant for small SoHo or branch offices. Should have tested on a Poweredge 2950 at least, with good storage.

Hyperthreading

They key thing I got from this was that VMWare Server not ESX doesn't seem to allow the virtual machine to take advantage of hyperthreading. With hyperthreading disabled, the performance of the virtual machine was MUCH closer to the performance of the real machine.

Having said that, as other posters have mentioned, this is with VMWare Server, not ESX which is the VMWare product meant for high load environments and not labs.

Re:VMware Server 1.0.1???

VMware Server 1.0.1 is their free virtualization product that runs on a host OS (linux or Windows). Most enterprises will use VMware ESX Server 3 with the VMware Virtual Infrastructure 3 series of products as it runs on "bare metal" and does not have the overhead of the host OS.

Re:Not a trusted source

The hyperthreaded capacity was actually 390 so a 3% gain.

Re:Not a trusted source

Since it's smaller than their error margin, not claiming it to be a tiny amount would mean to admit that there were large measurement errors. But in any case, being smaller than the error margin means that comparing the numbers is completely meaningless; in reality it might not have had any effect, or it might even have been a loss instead of a gain.

Re:Not a trusted source

it was the font size that was so tiny you couldn't even read the numbers correctly.

not your fault! ;)

Re:Not a trusted source

Mod me down!

It's a little early here and my vision was a bit blurry, it is 390 not 350!