Forgot your password?
typodupeerror
Operating Systems Software

Virtualizing Cuts Web App Performance 43% 223

Posted by kdawson
from the price-to-pay dept.
czei writes "This just-released research report, Load Testing a Virtual Web Application, looks at the effects of virtualization on a typical ASP Web application, using VMWare on Linux to host a Windows OS and IIS web server. While virtualizing the server made it easier to manage, the number of users the virtualized Web app could handle dropped by 43%. The article also shows interesting graphs of how hyper-threading affected the performance of IIS." The report urges readers to take this research as a data point. No optimization was done on host or guest OS parameters.
This discussion has been archived. No new comments can be posted.

Virtualizing Cuts Web App Performance 43%

Comments Filter:
  • Bogus Test (Score:5, Informative)

    by Anonymous Coward on Thursday March 29, 2007 @08:40AM (#18526831)
    Who uses VMWare Server in a production environment anyway? We run all of our Web services, Exchange servers and SQL databases in VMWare's Virtual Infrastructure 3. VMWare Player and Server are only ment for lab evironments and low load applications. VMWare even says as much on their website. Either this is just FUD or the author is an idiot. In other news water is wet.
  • by Fuyu (107589) on Thursday March 29, 2007 @08:41AM (#18526845)
    They performed the test on VMware Server not VMware ESX Server which is what most enterprises will use. VMware ESX Server runs on "bare metal", so it does not have the overhead of the host operating system.
  • Sounds about right (Score:2, Informative)

    by Anonymous Coward on Thursday March 29, 2007 @08:45AM (#18526889)
    My first attempt at virtualization was last September with VMWare Server. During testing everything seemed fine. When everything was using it, performance was awful. Everything crawled. I ended up doing an all-nighter to move everything back to a regular server. Note, I wasn't overloading things. There was only one VM on the host. The memory was fixed, not paged to a disk like it is by default. The hard drive was preallocated. My intention for virtualization was to make things easier to manage.

    That's when I started experimenting with Xen. This time I put the test under a very high load, and it seemed to handle everything well. I deployed it in October and so far there hasn't been a single performance issue.

    I'm now totally addicted to Xen. I create Vms all the time, have split up services into different VMs (ie, when cups crashes it no longer takes out the copy of samba that handles logins, damn I hate cups). So far, no performance issues at all.
  • by Fuyu (107589) on Thursday March 29, 2007 @08:48AM (#18526921)
    VMware Server 1.0.1 is their free virtualization product that runs on a host OS (linux or Windows). Most enterprises will use VMware ESX Server 3 with the VMware Virtual Infrastructure 3 series of products as it runs on "bare metal" and does not have the overhead of the host OS.
  • by dc29A (636871) on Thursday March 29, 2007 @08:50AM (#18526927)
    They performed the test on VMware Server not VMware ESX Server which is what most enterprises will use. VMware ESX Server runs on "bare metal", so it does not have the overhead of the host operating system.

    Doesn't VMWare ESX run on some modified Red Hat version?

    Also, we run ESX in our production environment, when we stress tested a web application running on IIS and with ASP/VB, the ESX machine couldn't give us more than 10 transactions per second (there was one single VM running on ESX). ESX was crawling.

    The same hardware running on Windows 2003 native gave us an easy 100+ without any problems. It seems that the overhead of ESX combined with huge number of context switches is what kills the performance of ESX. For non web applications like file servers, administrator consoles and whatnot, ESX is a beauty and great money saver.

    For web applications, I would avoid ESX like the black plague.
  • by dagenum (580191) on Thursday March 29, 2007 @08:52AM (#18526947)
    The hyperthreaded capacity was actually 390 so a 3% gain.
  • by Anonymous Coward on Thursday March 29, 2007 @08:52AM (#18526951)
    If eth0 is shared between host and guest OS and host OS is Linux:

    # ethtool -K eth0 tso off
  • by Fuyu (107589) on Thursday March 29, 2007 @08:57AM (#18526997)
    Yes VMware ESX Server runs a modified version of Red Hat Linux.

    According to Wikipedia [wikipedia.org], "VMware ESX Server uses a stripped-down proprietary kernel (derived from work done on Stanford University's SimOS [stanford.edu]) that replaces the Linux kernel after hardware initialization. The Service Console (also known as "COS" or as "vmnix") for ESX Server 2.x derives from a modified version of Red Hat Linux 7.2. (The Service Console for ESX Server 3.x derived from a modified version of Red Hat Enterprise Linux 3.) In general, this Service Console acts as a boot-loader for the vmkernel and provides management interfaces (CLI, webpage MUI, Remote Console). This VMware ESX hypervisor virtualization approach provides lower overhead and better control and granularity for allocating resources (CPU-time, disk-bandwidth, network-bandwidth, memory-utilization) to virtual machines. It also increases security, thus positioning VMware ESX as an enterprise-grade product."
  • Re:Bogus Test (Score:5, Informative)

    by Sobrique (543255) on Thursday March 29, 2007 @09:17AM (#18527161) Homepage
    Actually, the company I worked for 6 months back, one of the projects I was involved in was 'VMWare'. Production stuff running on on the ESX servers (which became 'virtual infrastructure') in our datacentre, as a cost effective scalable environment. Yes, we weren't getting 'uber performance' but then again, we were running 150 or so VMs on an 6 server VMWare farm.

    One of the other things we prototyped and deployed was 'site services packages' - get GSX (now VMWare Server), stick it on a pair of 2U servers, and attach a storage array to both of them. Then create your 'template' fileserver, DHCP server, print server, proxy, that kind of thing and deploy them to this package. It worked very well indeed - you get a whole new order of magnitude on stability (although to be fair that's in part because we through away the crappy workstations that were doing the 'low intensity' stuff) and was extremely managable, and trivially replacable in the event of a hardware failure.

    Performance? No, VMWare isn't that great on performance - whilst it's not bad, in an ideal situation, fundamentally what you are doing is introducing an overhead on your system. And probably contention too. But it's really good at efficient resource utilisation, easy manageability and maintainability.

    As an experienced sysadmin, my reaction is screw performance. Let's start with reliable and scalable, and then performance just naturally follows, as does a really high grade service.

    Proactive laziness is a fundamental of systems admin. Your job, is essentially to put yourself out of a job - or more specificially, free up your time to play with toys. The best way to do this is build something stable, well documented and easily maintainable. Then your day consists of interesting stuff, punctuated by the odd RTFM when something doesn't work quite right.

  • by TheRaven64 (641858) on Thursday March 29, 2007 @09:25AM (#18527247) Journal
    The biggest overhead from most forms of virtualisation is from emulated devices. If you have loads of money, you can give it to IBM and get some hardware with virtualisation-aware network and block device controllers. Then you get good performance. Alternatively, you can use paravirtualised device drivers. Xen supports this by default, and I think KVM does now for networks. Not sure about VMWare.

    With paravirtualised devices, or devices that are virtualisation-aware, a VM can be within 10% of the performance of a real machine quite easily. Without I'm surprised they even got to 57% of native performance for web applications.

  • Re:Pointless test? (Score:4, Informative)

    by LurkerXXX (667952) on Thursday March 29, 2007 @09:39AM (#18527395)
    No it's not insane. Lots of customers want full root access on their systems so they can install whatever they want (different database or other servers, or even alternate OS's). Virtualization is the only way to go for that.
  • by Sangui5 (12317) on Thursday March 29, 2007 @09:41AM (#18527413)
    It isn't surprising that VMWare would be bad at a web-app workload. See the original paper on Xen:

    http://www.cl.cam.ac.uk/research/srg/netos/papers/ 2003-xensosp.pdf [cam.ac.uk]

    Top of page 9 has a chart comparing native Linux, Xen, VMWare, and UML for different workloads. They show VMWare degrading performance by over 70% for SPECWEB 99.

    Web applications are OS intensive; while VMWare is quite good at pure CPU-bound tasks, it has to perform a lot of emulation whenever you are running inside the OS. So it will stink at anything with lots of small IO, lots of metadata operations, or lots of process creation/switching. For example, VMWare shows a whopping 90% slowdown for OLTP database workloads, according to the Xen paper, and it really isn't surprising. The OS microbenchmarks in the above paper (page 10) show that VMWare has abysmal performance for things like fork(), exec(), mmap(), page faults, and context switches.

    Basically, Xen doesn't have to emulate the OS, because they make modifications to the OS. VMWare does dynamic binary rewriting (think fancy emulation) to run an unmodified OS; they therefore pay through the nose in performance overhead for OS-intensive workloads.
  • by tepples (727027) <tepples&gmail,com> on Thursday March 29, 2007 @09:49AM (#18527515) Homepage Journal

    Not to mention the fact that paravirtualization as well as hardware assisted virtualization like Xen offers (and later Longhorn) really cut the performance issues WAY the hell down.
    Paravirtualization also requires a free software OS kernel, and IIS-only web applications are do not yet run on any free software kernel. (ReactOS is nowhere near mature enough.) Any virtualization also requires more OS licenses and higher-class, more expensive OS licenses. Or do you claim that all web app developers should drop IIS-only frameworks immediately, and all enterprises that rely on IIS-only web applications should drop their mission-critical IIS-only web applications immediately?
  • by Anonymous Coward on Thursday March 29, 2007 @09:50AM (#18527525)
    ESX Server still gives you a base 40% performance hit. I run a ~600 VM farm under VI3 and our performance on Apache fell from 15000 requests/s (mostly static content) to 5000. That was during a load test with one single virtual machine running on the blade. The same load test using IIS went from 13000 to 9000. Also a huge performance hit, although not quite as bad as on Linux. And before anyone says anything, I'm a linux tech and I was somewhat deprssed about the results, to our windows techs great joy.
  • by Natales (182136) on Thursday March 29, 2007 @09:55AM (#18527601)
    VMware's vmxnet driver is paravirtualized and it does provide better performance than the traditional pcnet32 virtual device driver, which operates 100% on software to maintain compatibility with other OSs.

    Regarding paravirtualization, it's already known that the new VMware Workstation 6 (currently in beta) and presumably the next version of VMware Server, will support VMware's version of paravirtualization called VMI, which was officially accepted as part of the stock Linux kernel starting on 2.6.21. This may help boosting the performance of Linux-based VMs significantly, and unlike the Xen version, it will boot a single kernel image, regardless of the physical or virtual underlying hardware platform.
  • Re:Pointless test? (Score:4, Informative)

    by GiMP (10923) on Thursday March 29, 2007 @10:04AM (#18527697)
    AssignUserId only works with the perchild MPM, which has the following caveat: "This module is not functional. Development of this module is not complete and is not currently active. Do not use perchild unless you are a programmer willing to help fix it."

    Thus, AssignUserId should NOT be used. SuExec can be used, of course, but that has its own limitations.

    Personally, I give users their own Apache processes on their own port (>1024) and use a reverse proxy. I make a living on it.
  • really bad report... (Score:1, Informative)

    by Anonymous Coward on Thursday March 29, 2007 @10:13AM (#18527815)
    But it highlights one thing: if you hand virtualization to clueless people, you'll get bad perfs.

    It also shows, both in the article and in the comments here, the severe misunderstanding surrounding the concept of "virtualization".

    I see lots of clueless people saying "uh, of course, virtualization perfs sucks". I think those people don't realize today's virtualization technology ain't grandpa's past-century emulators.

    There are today virtualization technologies that offer basically native speeds. Xen can now run in two modes (para-virt or hardware-virt, the latter if the MOBO/BIOS/CPU supports Intel-VT / AMD-V)... In paravirt mode Xen offers native speeds (the overhead is so small you'll have a hard-time measuring it). Better: network I/O ain't good enough for you? Simply "passthrough" a PCI device (say a PCI network card) to your paravirtualized guest. The guest (and only the guest) is directly accessing the PCI card (no more network I/O problems). But you can't run Windows on Linux using paravirt under Xen...

    In hardware-virtualized mode, under Xen (or KVM, which only does hardware-virt), you can run Windows. Network and disk I/O, for hardware virt, at this point sucks. However you can install special drivers in your guest to make it speedier (drivers for Windows under Xen are $$$ and under development for KVM).

    But, wait, there's more to come... Next gen IOMMU is around the corner. And as soon as it gets implemented in Xen, the already super-fast virtualized system gets an additional boost and you'll have something even closer to native, even when running Windows under Xen.

    If you think "virtualization will always be slower" you need a reality check: the CPU makers are working hard so that the virtualization overhead becomes irrelevant. And suddenly the ones not using virtualization will find themselves with a less capable, less secure, less maintanable box being, in some particular, anecdotical, cases only 0.05% faster.

    Virtualization is here to stay and the overhead, already very small today, will keep shrinking.

  • Re:Pointless test? (Score:4, Informative)

    by Just Some Guy (3352) <kirk+slashdot@strauser.com> on Thursday March 29, 2007 @11:03AM (#18528473) Homepage Journal

    You run virtualised web servers because 99.9% of all web servers are idle at any given time. So you put 100 on a server.

    If you have a real need to run 100 separate Apache instances, then you'll want something much higher-level than VMWare. For us, that would be a FreeBSD jail, where each instance would get its own chrooted home directory and IP address. That way, you're not allocating resources to 100 little-used OS images; each shares from the same memory and hard drive pool. Jails are slightly limited in that I'd like a way to limit CPU and memory allocation, but in practical application this really works very well today.

  • Re:Bogus Test (Score:5, Informative)

    by Bohiti (315707) on Thursday March 29, 2007 @11:59AM (#18529331) Homepage
    Actually, as it's been explained to me, the ESX hypervisor itself is pure proprietary code (and small, too). The Service Console is very readily admitted to be a tweaked out RHEL (3, I believe..). Linux is used to boot, and then (magically?) transfers control of the bare metal to the hypervisor. Linux then jumps into a virtual machine, although it's not presented like a virtual machine, which creates all this confusion.

    In the end, the tweaked RHEL that you interact with (ssh, scp) is not the hypervisor, but a VM with special tools that can manipulate the hypervisor.
  • Re:Well, (Score:3, Informative)

    by inKubus (199753) on Thursday March 29, 2007 @12:39PM (#18529887) Homepage Journal
    It's not just support reasons. A lot of MSFT products require a dedicated server because they use the Default Web Site in IIS ;) Multi-Tenancy is not an option for many even modern server products. So, virtualize the server.
  • Re:Pointless test? (Score:3, Informative)

    by Just Some Guy (3352) <kirk+slashdot@strauser.com> on Friday March 30, 2007 @09:51AM (#18542555) Homepage Journal

    The chroot-jail based hosting system I used had a number of problematic limitations: I couldn't install any software that needed to run as root (e.g. to open ports below 1024), I couldn't change configuration of a variety of services that would open security problems for the hosting provider (e.g. the mail server, I think they were using exim), users running processes on the same server that consumed too much CPU and or memory had too much negative impact on my server's performance.

    Of those, only the last is relevant to FreeBSD jail setups. If I created a jail for you and gave you root, you would be root, full stop. The only things you could do would be install your own kernel (since only one kernel - that of the host OS - is running). We use them to virtualize multiple distinct systems on the same hardware, with the idea that the mailserver always runs under a light load and doesn't interfere with the database server hosted on the same machine.

    There's a decent Wikipedia article [wikipedia.org] on the subject, even if it kind of comes across like an advertisement. In short, it sounds like your hosting provider ran a bad server. Don't extrapolate their incompetence to the general state of the art.

    VMWare ESX server *can* share memory and free disk space between instances. Multiple instances can even have the same page of physical memory mapped into them, so running multiple identical servers doesn't actually take more memory than it does on a single instance (although there are overheads).

    Assuming a strategy like copy-on-write, I can understand how two instances started from the same configuration could begin with most of their memory shared. However, it seems like that would eventually become a tiny percentage of their actual address space as processes start and die, allocate memory and free it, etc. I mean, to the best of my knowledge, when Unix fork()s a process, it doesn't keep track of when it can later re-merge the address space of the parent and child. They may start as identical copies, but pretty soon their data segments will be completely different. In the case where each process is actually a virtualized system where the data segment is hugely bigger than the shared code, I'd think that would happen pretty quickly.

After an instrument has been assembled, extra components will be found on the bench.

Working...