Virtualizing Cuts Web App Performance 43% 223
czei writes "This just-released research report, Load Testing a Virtual Web Application, looks at the effects of virtualization on a typical ASP Web application, using VMWare on Linux to host a Windows OS and IIS web server. While virtualizing the server made it easier to manage, the number of users the virtualized Web app could handle dropped by 43%. The article also shows interesting graphs of how hyper-threading affected the performance of IIS." The report urges readers to take this research as a data point. No optimization was done on host or guest OS parameters.
Bogus Test (Score:5, Informative)
This is VMware Server and not ESX Server (Score:5, Informative)
Sounds about right (Score:2, Informative)
That's when I started experimenting with Xen. This time I put the test under a very high load, and it seemed to handle everything well. I deployed it in October and so far there hasn't been a single performance issue.
I'm now totally addicted to Xen. I create Vms all the time, have split up services into different VMs (ie, when cups crashes it no longer takes out the copy of samba that handles logins, damn I hate cups). So far, no performance issues at all.
Re:VMware Server 1.0.1??? (Score:2, Informative)
Re:This is VMware Server and not ESX Server (Score:2, Informative)
Doesn't VMWare ESX run on some modified Red Hat version?
Also, we run ESX in our production environment, when we stress tested a web application running on IIS and with ASP/VB, the ESX machine couldn't give us more than 10 transactions per second (there was one single VM running on ESX). ESX was crawling.
The same hardware running on Windows 2003 native gave us an easy 100+ without any problems. It seems that the overhead of ESX combined with huge number of context switches is what kills the performance of ESX. For non web applications like file servers, administrator consoles and whatnot, ESX is a beauty and great money saver.
For web applications, I would avoid ESX like the black plague.
Re:Not a trusted source (Score:3, Informative)
Re:This has been my experience too (Score:1, Informative)
# ethtool -K eth0 tso off
Re:This is VMware Server and not ESX Server (Score:2, Informative)
According to Wikipedia [wikipedia.org], "VMware ESX Server uses a stripped-down proprietary kernel (derived from work done on Stanford University's SimOS [stanford.edu]) that replaces the Linux kernel after hardware initialization. The Service Console (also known as "COS" or as "vmnix") for ESX Server 2.x derives from a modified version of Red Hat Linux 7.2. (The Service Console for ESX Server 3.x derived from a modified version of Red Hat Enterprise Linux 3.) In general, this Service Console acts as a boot-loader for the vmkernel and provides management interfaces (CLI, webpage MUI, Remote Console). This VMware ESX hypervisor virtualization approach provides lower overhead and better control and granularity for allocating resources (CPU-time, disk-bandwidth, network-bandwidth, memory-utilization) to virtual machines. It also increases security, thus positioning VMware ESX as an enterprise-grade product."
Re:Bogus Test (Score:5, Informative)
One of the other things we prototyped and deployed was 'site services packages' - get GSX (now VMWare Server), stick it on a pair of 2U servers, and attach a storage array to both of them. Then create your 'template' fileserver, DHCP server, print server, proxy, that kind of thing and deploy them to this package. It worked very well indeed - you get a whole new order of magnitude on stability (although to be fair that's in part because we through away the crappy workstations that were doing the 'low intensity' stuff) and was extremely managable, and trivially replacable in the event of a hardware failure.
Performance? No, VMWare isn't that great on performance - whilst it's not bad, in an ideal situation, fundamentally what you are doing is introducing an overhead on your system. And probably contention too. But it's really good at efficient resource utilisation, easy manageability and maintainability.
As an experienced sysadmin, my reaction is screw performance. Let's start with reliable and scalable, and then performance just naturally follows, as does a really high grade service.
Proactive laziness is a fundamental of systems admin. Your job, is essentially to put yourself out of a job - or more specificially, free up your time to play with toys. The best way to do this is build something stable, well documented and easily maintainable. Then your day consists of interesting stuff, punctuated by the odd RTFM when something doesn't work quite right.
Re:single data point is correct (Score:5, Informative)
With paravirtualised devices, or devices that are virtualisation-aware, a VM can be within 10% of the performance of a real machine quite easily. Without I'm surprised they even got to 57% of native performance for web applications.
Re:Pointless test? (Score:4, Informative)
These results are pretty much as expected (Score:3, Informative)
http://www.cl.cam.ac.uk/research/srg/netos/papers
Top of page 9 has a chart comparing native Linux, Xen, VMWare, and UML for different workloads. They show VMWare degrading performance by over 70% for SPECWEB 99.
Web applications are OS intensive; while VMWare is quite good at pure CPU-bound tasks, it has to perform a lot of emulation whenever you are running inside the OS. So it will stink at anything with lots of small IO, lots of metadata operations, or lots of process creation/switching. For example, VMWare shows a whopping 90% slowdown for OLTP database workloads, according to the Xen paper, and it really isn't surprising. The OS microbenchmarks in the above paper (page 10) show that VMWare has abysmal performance for things like fork(), exec(), mmap(), page faults, and context switches.
Basically, Xen doesn't have to emulate the OS, because they make modifications to the OS. VMWare does dynamic binary rewriting (think fancy emulation) to run an unmodified OS; they therefore pay through the nose in performance overhead for OS-intensive workloads.
IIS can't be paravirtualized (Score:2, Informative)
Re:This is VMware Server and not ESX Server (Score:3, Informative)
Re:single data point is correct (Score:2, Informative)
Regarding paravirtualization, it's already known that the new VMware Workstation 6 (currently in beta) and presumably the next version of VMware Server, will support VMware's version of paravirtualization called VMI, which was officially accepted as part of the stock Linux kernel starting on 2.6.21. This may help boosting the performance of Linux-based VMs significantly, and unlike the Xen version, it will boot a single kernel image, regardless of the physical or virtual underlying hardware platform.
Re:Pointless test? (Score:4, Informative)
Thus, AssignUserId should NOT be used. SuExec can be used, of course, but that has its own limitations.
Personally, I give users their own Apache processes on their own port (>1024) and use a reverse proxy. I make a living on it.
really bad report... (Score:1, Informative)
It also shows, both in the article and in the comments here, the severe misunderstanding surrounding the concept of "virtualization".
I see lots of clueless people saying "uh, of course, virtualization perfs sucks". I think those people don't realize today's virtualization technology ain't grandpa's past-century emulators.
There are today virtualization technologies that offer basically native speeds. Xen can now run in two modes (para-virt or hardware-virt, the latter if the MOBO/BIOS/CPU supports Intel-VT / AMD-V)... In paravirt mode Xen offers native speeds (the overhead is so small you'll have a hard-time measuring it). Better: network I/O ain't good enough for you? Simply "passthrough" a PCI device (say a PCI network card) to your paravirtualized guest. The guest (and only the guest) is directly accessing the PCI card (no more network I/O problems). But you can't run Windows on Linux using paravirt under Xen...
In hardware-virtualized mode, under Xen (or KVM, which only does hardware-virt), you can run Windows. Network and disk I/O, for hardware virt, at this point sucks. However you can install special drivers in your guest to make it speedier (drivers for Windows under Xen are $$$ and under development for KVM).
But, wait, there's more to come... Next gen IOMMU is around the corner. And as soon as it gets implemented in Xen, the already super-fast virtualized system gets an additional boost and you'll have something even closer to native, even when running Windows under Xen.
If you think "virtualization will always be slower" you need a reality check: the CPU makers are working hard so that the virtualization overhead becomes irrelevant. And suddenly the ones not using virtualization will find themselves with a less capable, less secure, less maintanable box being, in some particular, anecdotical, cases only 0.05% faster.
Virtualization is here to stay and the overhead, already very small today, will keep shrinking.
Re:Pointless test? (Score:4, Informative)
If you have a real need to run 100 separate Apache instances, then you'll want something much higher-level than VMWare. For us, that would be a FreeBSD jail, where each instance would get its own chrooted home directory and IP address. That way, you're not allocating resources to 100 little-used OS images; each shares from the same memory and hard drive pool. Jails are slightly limited in that I'd like a way to limit CPU and memory allocation, but in practical application this really works very well today.
Re:Bogus Test (Score:5, Informative)
In the end, the tweaked RHEL that you interact with (ssh, scp) is not the hypervisor, but a VM with special tools that can manipulate the hypervisor.
Re:Well, (Score:3, Informative)
Re:Pointless test? (Score:3, Informative)
Of those, only the last is relevant to FreeBSD jail setups. If I created a jail for you and gave you root, you would be root, full stop. The only things you could do would be install your own kernel (since only one kernel - that of the host OS - is running). We use them to virtualize multiple distinct systems on the same hardware, with the idea that the mailserver always runs under a light load and doesn't interfere with the database server hosted on the same machine.
There's a decent Wikipedia article [wikipedia.org] on the subject, even if it kind of comes across like an advertisement. In short, it sounds like your hosting provider ran a bad server. Don't extrapolate their incompetence to the general state of the art.
Assuming a strategy like copy-on-write, I can understand how two instances started from the same configuration could begin with most of their memory shared. However, it seems like that would eventually become a tiny percentage of their actual address space as processes start and die, allocate memory and free it, etc. I mean, to the best of my knowledge, when Unix fork()s a process, it doesn't keep track of when it can later re-merge the address space of the parent and child. They may start as identical copies, but pretty soon their data segments will be completely different. In the case where each process is actually a virtualized system where the data segment is hugely bigger than the shared code, I'd think that would happen pretty quickly.