Performance Evaluation of Xen Vs. OpenVZ 116
An anonymous reader writes "Compared to an operating-system-level virtualization technology like OpenVZ, Xen — a hypervisor-level virtualization technology that allows multiple operating systems to be run with and without para-virtualization — trades off performance for much better isolation and security. OpenVZ's performance advantage due to running virtual containers in a single operating system kernel can be significant. A performance evaluation study (PDF) done by researchers at the University of Michigan and HP labs provides insight into how big a performance penalty Zen pays and what causes the overheads (primarily L2 cache misses)." From the report: "We compare both technologies with a base system in terms of application performance, resource consumption, scalability, low-level system metrics like cache misses and virtualization-specific metrics like Domain-0 consumption in Xen. Our experiments indicate that the average response time can increase by over 400% in Xen and only a modest 100% in OpenVZ as the number of application instances grows from one to four... A similar trend is observed in CPU consumptions of virtual containers."
Better known as Virtuzzo (Score:1)
Yes, but does it run Linux\\\\\Windows ? (Score:1, Offtopic)
But which ones of these things can run Windows clients, at least XP? VMWare can, User Mode Linux
Re: (Score:3, Informative)
Re: (Score:1)
> VMWare can, User Mode Linux can't, but what about OpenVZ, Xen, and some of the others?
Xen can run Windows without para virtualization using new processor VT extensions. OpenVZ's commercial implementation Virtuozzo can run Windows on Linux.
> do any of these make USB devices visible to the client OS?
> Or do they all just have to network-mount resources that are actually mounted in the host OS?
Both Xen, VMware and OpenVZ all al
VMWare is who they are going after (Score:2)
I am a big fan of Virtuozzo and OpenVZ. I just wish vzstat would be included with the free stuff.
Stop the press (Score:3, Insightful)
No kidding, that's why I use it! Xen's performance ain't so bad. Show me a better performing virtualization solution that matches or bests Xen's isolation security - then we'll talk.
Re: (Score:2)
Re:Stop the press (Score:5, Insightful)
So to sum, we've got OpenVZ, Virtualbox, KVM/Qemu, Xen, VMWare, Virtual Iron, and Virtuozzo. With so much virtualization software, I personally think performance takes a back seat to functionality (sure OpenVZ is fast, cool, what will it do for me that VMWare or Xen won't?). Is there really that much space in the virtualization landscape?
Re: (Score:3, Interesting)
Re: (Score:1)
Nice to see you pimping xen-tools [xen-tools.org] :)
Re: (Score:2)
Now, you see, I've actually used xen-tools and I notice that you've cunningly left out the part where you edit
because what you get when you *don't* edit that config file to suit is this (for example):
The kernel image we're trying to use does not exist.
The image is -
Aborting
ie the default config file for the current version of xen-tools points to the wrong kernel. And thats just for
Re: (Score:2)
Looks quite easy to me.
That's because you're not doing anything interesting.
Try working with multiple vlans, bonded interfaces, multiple drives in VMs, SANs, etc, then come back.
Re: (Score:2)
What drives me up the wall at the moment is bonded interfaces. The Xen scripts flat out suck for doing any sort of ethernet bonding in domain0.
Re:Stop the press (Score:5, Interesting)
Yeah but Xen is still a royal PITA to get running. KVM wasn't bad, and VMWare was pretty easy. I haven't even seen OpenVZ.
Yes, Xen is harder to install. But to compare it with KVM ... did you try to use them? KVM (at least, last time I tried -- which was only a couple weeks ago) is still in development, and the performance is so low compared to Xen that it's not even funny.
On the other hand, VMware is very nice, specially the free Server edition, and it's really easy to use. But even so, performance is better in Xen. Check this [xensource.com]. Paravirtualization needs modified guests, but the outcome is so good that VMware is trying paravirtualization too [vmware.com].
VMWare is so far ahead it will take some time for Xen to be considered out of the hobbyist market and in the commercial one
What do you think is needed for Xen to be considered apt for commercial use? Remember that Xen can use unmodified guests if the hardware supports VTX/SVM instructions, which means that it can run Windows. Pretty front-ends? Xensource (which is slashdotted now, I guess, because it times out from here) offers one, and you also have Enomalism [enomalism.com].
Besides, by what Wikipedia says about OpenVZ [wikipedia.org], it seems to be more a solution like jails [wikipedia.org], because it uses the same kernel for both the host and the guest systems. The phrase "glorified chroot" comes to mind, though I'm aware that it's more than that (just adding it for the sake of trolling, I guess :-)). Xen, VMware and QEMU/KVM are, on the other hand, real virtualization solutions, where all the virtual system runs completely isolated.
I wouldn't recommend Xen for home use (VMware Server is a better and easier option, IMHO), but saying that it's not ready and comparing it to QEMU/KVM is almost a joke.Re:Stop the press (Score:4, Informative)
VMWARE is an excellent DR/BC solution due to complete virtual machine portability. However consolidation ratios (8:1 maybe) and performance are not that great, for consolidation you've probably only saved a few Us, but you have not really helped reduce the support burden, you'll still need lots of middleware / server management software to manage your estate. To actually get the most out of the DR/BC solution you also need an FC SAN and the complete Virtual Centre suite. If you've that kind of money to throw around then clearly you are in it for BC/DR purposes and VMWare is the right option. At this level you don't care if your consolidation ratios are 1:0.9 or worse.
Virtuozzo/VZ on the other hand wins hands down for consolidation and management; you can easily fit 20-30 or more VPSes on a single server; and whatever the consolidation ratio the responsiveness will always be better than the same server under VMWare. However there are some drawbacks, Virtuozzo doesn't give you complete portability; you can only host Windows 2003 guest servers on a Windows 2003 hardware node for instance and you are unable to install device drivers. Adding a new VPS takes about as long and uses about as much resource as creating a new user on a system.
I use both solutions; each definitely have their own place and couldn't be more different in their relative strengths.
Jason
Re: (Score:1)
Re: (Score:3, Informative)
Jason.
Re: (Score:2)
Re: (Score:2)
Unless you're spending $20k-$50k per server on hardware, VMWare pricing will seem very expensive (in the $3-$4k range per server is what I was quoted a few months ago). For smaller shops that have a few $6k-$10k servers and a small SAN, the VMWare costs make it a very hard sell (almost better to order another server).
Re: (Score:2)
OpenVZ is easy to setup (once you get the kernel and packages setup, which is a breeze in Debian 3.1 and 4.0). It runs ridiculously fast, it consumes little memory, and takes little disk space (which can be furthe
Re: (Score:2)
Yeah but Xen is still a royal PITA to get running.
If you think that Xen is a "royal PITA" to get running, then how do you manage to work with a server? There is a slight learning curve, but after that it's extremely easy to deal with. Much like most Unix things. The massive performance benefit that Xen gives over VMWare Server (the only one I use) is well worth the hour it takes to learn the basics.
Now, if you're virtualizing a desktop system, VMWare rocks. I use it daily for this. But after using both in production, Xen clearly has a significant perf
Re: (Score:2)
If you think that Xen is a "royal PITA" to get running, then how do you manage to work with a server?
Most likely, he's trying to do non-trivial things with Xen. Stuff like bridges to multiple vlans, interface bonding, multiple drives, and the like.
Xen *sucks* from an administrative perspective if your environment is anything remotely complicated. VMWare has nothing to fear.
There is a slight learning curve, but after that it's extremely easy to deal with. Much like most Unix things. The massive perfor
Re: (Score:1)
OS-based virtualization, which is done by OpenVZ, is based on "Virtual Environments" that all run under one kernel. It's kinda sorta like running different servers in their own "chroot jail", but with their own virtual interfaces, their own quotas, etc. etc. Generally because you're not semi-emulating hardware paravirtualization is faster
Para-virtualization, which is done by Xen, K
Re: (Score:2)
Repeat.
Granted, I only used it for installing other FC6 images, but it was very simple.
Re: (Score:2)
Re: (Score:1)
Re: (Score:1)
> Show me a better performing virtualization solution that matches or bests Xen's isolation security - then we'll talk.
KVM [qumranet.com] ? Though it is in early stages of development, in theory, it may be able to run faster than Xen. It's goals, however are different from that of Xen's
Re: (Score:1)
Xen for that matter any virtualization will run CPU intensive applications very well and close to native. If it doesn't, there is some thing really wrong. The problem happens in I/O intensive applications. It's not our (I am the primary author of the report) goal to find clever scenarios to find where Xen is worse. These scenarios are real and use complex multi-tier setups. These scena
KVM (Score:2, Insightful)
Looks like analyst talk (Score:3, Funny)
I don't know about you but it still makes my eyes hurt!
Re:Looks like analyst talk (Score:4, Informative)
Hypervisor [wikipedia.org] — the software that makes the virtualization happen... sometimes means virtualization that runs on bare-metal, rather than under a host OS.
Paravirtualization [wikipedia.org] — I think this just refers to the cases where the guest OS is modified/recompiled to run work without needing to run in Ring 0, and instead changes those to be explicit calls to the virtualization software.
So translated, I think that means "virtualization software that runs on bare-metal, both using unmodified guest OS's, and modified guest OS's."
Re: (Score:3, Insightful)
From everything I can see though the word is useless and it amounts to the equivalent of computer scientists being fussy. VM or VM OS are better choices.
Re: (Score:2)
> I don't know about you but it still makes my eyes hurt!
Really, it's not that complex. The technology monitors all system calls, and makes a judgement call - if it's safe to let it through, it routes it through the EPS conduits. If it's not, it routes it through the GNDN tubes. As long as you don't overload the EPS taps, it's all good.
Oblig. Nonsensical reference (Score:5, Funny)
Zen's performance issues were fixed by Avon, under Orac's guidance.
Re: (Score:2)
Re: (Score:1, Offtopic)
"'Twas brillig, and the slithy tovesDid gyre and gimble in the wabe:All mimsy were the borogoves,And the mome raths outgrabe".
Sorry... it was the 1st thing I thought of when I read the article
Re: (Score:2)
Re: (Score:2)
I must be a sad, old geek ... (Score:1)
Oh dear!
vserver + unionfs (Score:1, Informative)
I can add and remove (semi) virtual machine at will. Each VM feels barely heavier than just an ordinary process.
I take my normal mount points, and make it the read-only layer. I then add a writable layer on top of that and that's it. I've also created some handy scripts that'll let me manage, add, remove, start, stop, etc, VM's.
And OpenVZ works with FreeBSD? Windows? (Score:1, Interesting)
Does anyone have links to show the support?
Re: (Score:2)
OpenVZ and Virtuozzo rely upon Linux kernel modifications; in other words, no FreeBSD, no Windows.
Both are good. (Score:5, Informative)
OpenVZ is ok if all of your child environments run the same OS and you don't care about them stealing each other's resources. We constantly have problems with customers overloading their VPS and causing problems with the other environments, this doesn't happen with Xen. I've fork bombed child environments and caused the load to spike to over 700 until it crashed, dom0 and the rest of my domUs just kept running like nothing was even happening.
OpenVZ also wins if you want to oversell hardware, Xen doesn't have "burstable" memory like OpenVZ does. Personally I prefer Xen for the jailing that it does and you can also run multiple OSes at the same time. I have a server at work that's running CentOS, Windows 2003, Windows Longhorn, and Gentoo all at the same time, OpenVZ only lets you run Linux on Linux.
Re:Both are good. (Score:5, Informative)
The big difference between Xen and OpenVZ comes down to what openvz calls "privvmpages" - memory that is claimed by running processes but not actually used.
The example here is this: two OpenVZ virtual environments (VE) can be set up on a server with, say, 1 gig of ram, with a gig of swap underneath it (So, RAM+Swap equals 2 gigs).
Those two virtual environments can be "oversold" in the amount of privvmpages they're allowed to use, because processes ask for more memory than they _actually use_ all the time. So let's say we give those two VEs 1.5 gigs of privvmpages (total of 3 gigs - more than RAM+Swap), but we only give them each 500 megs of oomguarpages. (less than RAM+Swap).
The thing that _should never_ go over RAM+Swap is oomguarpages (out of memory guaranteed pages) - pages of memory that are guaranteed (OpenVZ measures some of its resources in pages and some in megs).
With Xen, on that server I just described you're locked in - there's no bursting, and there's no dynamic allocation going on. You give one domU 750 megs of ram and you give the other domU 750. That's it.
Re: (Score:3, Insightful)
Remind me to never EVER hire you to do any work for me. Ever heard of "the right tool for the job"? I am a huge fan of virtualisation, and have been using it for years, VMWare mostly, but lately more and more with Xen. Given that Xen simply does not yet play nice with most of the lower cost hardware, and has several significant shortcomings in real-life enterprise production environments (running
Re: (Score:1)
I wasn't specifically saying I would do the virtualization with Xen... if it was specifically a Linux server I would most definitely run it in an OpenVZ VE. You seem to think "virtualization" automatically means paravirtualization, not OS-based virtualization.
What I meant was that even
Re: (Score:2)
Re: (Score:1)
OpenVZ isolates VEs from each other pretty good, unless you misconfigure it. The problem here is you can't configure the system properly (i.e. you oversell way too much). See this article [openvz.org] to get the details on how to configure your system in a proper manner.
Re: (Score:2)
Logged in one day and nothing worked. The host had changed to Virtuozzo. They tried to sell us on the advantages 'look it has a cool web based frontend' they said. It sucked. It had lower bandwidth, lower memory available and was as slow as molasses (like the load average never went below 4 on an idle system running nothing else).
We coped with that for abou
Re: (Score:1)
Hear, hear (Score:1)
Thnaks for OpenVZ btw! It's a great product, when used correctly
Re: (Score:2)
OpenVZ also has a very distinct advantage - all processes in guest VEs are visible to the tools on the main host.
Yes, but ... (Score:4, Funny)
Re:Yes, but ... (Score:4, Funny)
And... (Score:5, Insightful)
Re: (Score:3, Interesting)
How should someone who is satisfied witn VMWare decide whether an alternative would be an improvement? When the license terms for VMWare prohibit any benchmarking its kind of hard to make a decision.
Re:And... (Score:4, Insightful)
Do I get a Karma bonus for conceding someone's point?
Re: (Score:2)
Meaning that, any methodology which favours anything other than vmware will not be approved. In this case, i think the lack of available benchmarks says more.
Xen vs VMware - personal experience (Score:5, Informative)
1. Cost - no contest, xen wins hands down $0 vs $5000/cpu.
2. Performance - xen wins noticably, i can get away with running 8 virtual machines with 1/4 the hardware that VMWare required for 6.
3. Capabilities - VMware versions ESX GSX and up beat Xen in ease of use and flexibility, anything less and Xen wins. Xen does have a quicker live migration capability, but falls short on conencting external hardware to the virtual machine (something that is trivial in VMware).
4. Stability - about even, maybe Xen. I've seen ESX crash once, and have never seen Xen crash.
5. Ease of use - VMWare no question. Theres a learning curve to Xen and setting up new VM images is a pain. (which is disappearing) There are fantastic tools for VMware that will let you manage virtual machine creation and even migration from a physical box. (some of which can be used to feed xen too =))
The bottom line. A Xen setup (using open source version) that can easily run 12 VMs costs about $2,000 on DIY hardware. A VMWare server/software combo to do the same? about $30,000. Assume an additional 40-80 hours of learning to get up to speed with Xen vs VMware.
The comments above cover the open source / free version of Xen. I have no experience with the commercial Xen offerings.
Personally i'm Xen biased due to cost.
Re: (Score:2)
Re: (Score:2)
Why would vmware compare xen to an old version of vmware? That seems rather fishy... You'd think they would want to compare xen to their latest and greatest
Re: (Score:3, Insightful)
Personally i'm Xen biased due to cost.
I agree. Just a little addition: besides, VMware licenses are even more important when you want to scale. Say you want to use two quad boxes for load balancing, running VMware in each: that's about 8 x 5000 = $40000 in licenses (or $35000 if the first CPU is for free; I'm using your prices, but I knew about a similar project where VMware was decided against due to licensing costs too). You can buy more hardware with that money and train all your staff to understand
Re: (Score:2)
Tell me more, please. Looking at the Xen performance paper, it looks like Xen & VMWare are pretty close, performance-wise. The most they ever show is a 23% boost on a couple tests - nowhere near the "1/4 the hardware". Not trying to bait, but honestly curious.
Re: (Score:2, Insightful)
There's one item you didnt list, though it might be folded under capabilities or ease of use...
Manageability is a key item for these types of setups. As people being to depend more and more on virtualization technologies (whatever those might be) and roll out virtualized production environments, you have to figure out how to monitor and manage them. This is one area where VMWare currently has the upper hand, as Xen's management API has not yet been stabilized.
The reason this is important is because with all
Re: (Score:3, Insightful)
Quite simply, a corporation is going to buy VMWare Virtual Infrastructure. So the performance isn't as good? So what? Throw hardware at it. It works.
We have a significant investment in VMWare VI3 where I work, and it's great. We run it on high-
Re: (Score:2)
Um, what about Redhat and Novel (SuSE)? Redhat have restructured how they sell RHEL 5 based exclusively on how many Xen VMs you're allowed run. Their standard RHEL 5 Server allows you to run up to 4 guest VMs, where as RHEL 5 Advanced Platf
Re: (Score:2)
Um, what about Redhat and Novel (SuSE)? Redhat have restructured how they sell RHEL 5 based exclusively on how many Xen VMs you're allowed run. Their standard RHEL 5 Server allows you to run up to 4 guest VMs, where as RHEL 5 Advanced Platform allows an unlimited number of guest VMs and also includes the RHEL Cluster suite. If the guest VMs are RHEL then it's all included in the one cost, and of cause it's fully supported. There's no excuse for turning your back on Xen based on support.
And who does one turn to when a bug is found in the code? One that is critical for your functionality?
Don't underestimate this as a driving factor for corporations. We had this exact problem with the first release of VI3, and it was a show-stopper bug. We called EMC, and had a significant patch sent out to us within 48 hours which was subsequently rolled up into the next release. The problem corporations see with Xen and OpenVZ is that there are no programmers at the companies you mentioned actually coding
Re: (Score:2)
Going off on a slight tangent, VMware Workstation (non-free, $$) is the only one that has a glimmer of (imperfect) DirectX support, although there are rumors about similar support in the forthcoming OS X version. This is important for home users (not enterprise or business users) that run windows to play real-time 3D games. ;-)
Re:And... (Score:4, Informative)
This is a bit out of date, but here is some comparison between Xen, User Mode Linux, and Vmware 3.2 (which is the most recent version that allows publication of benchmarks).
http://www.cl.cam.ac.uk/research/srg/netos/xen/pe
Xen is always faster than Vmware, with the exact amount varying depending on the specific load. They've all improved since then, of course.
Re: (Score:1, Interesting)
(which is the most recent version that allows publication of benchmarks).
Really ? Then what is this:
http://blogs.xensource.com/rogerk/wp-content/uploa ds/2007/03/hypervisor_performance_comparison_1_0_5 _with_esx-data.pdf [xensource.com]
Xensource themselves claim "xensource performance as well as vmware" in thier white paper which shows VMware to be slower only in specjbb and xensource is slower in some areas. And this is the highly optimized commercial version (not free, thousand something per core - still cheaper than vmware though)
Re: (Score:1)
Re: (Score:1)
Nothing like a virtualization comparison that ignores the 800 guerilla that is VMWare. How do the learning curves, performance and security of these products compare with VMWare? Why should someone who is satisfied with VMWare consider other alternatives?
We would gladly run the same kind of experiments on VMware, but VMware does not allow publishing results, unless they have a look at them first. They probably don't want unfavourable results to be published. As an academic, I really hate such restrictions and want to do an un-biased study.
Pradeep (Primary author of the mentioned report)
Re: (Score:1)
The article (yup, I've speed read TFA) defines Xen as a para-virtualization systems... Completely ignoring that since now years Xen also allows to do hardware-virtualization. Another posted also noted already that Xen's power lies in its ability to run different OSes.
Btw I happen to run my Samba / NFS / CVS / SVN server on a Xen para-virtualized domU. For hardware-virtualization I tried Xen too and the open, free, version lacks good I/O drivers for Windows (slow network and slow disk).
We very well know that Xen supports hardware-virtualization as well. As you pointed out, the performance is abysmal because Xen still uses IO emulation while using hardware virtualization.
People want to Google on exactly "Which virtualization is right for you" and read infos from a knowledgable sysadmin (managing thousands of servers and virtual servers on a lot of various platforms and, no, it's not me) instead of an article performing micro-benchmarking of hypercalls to compare apples to oranges.
These are NOT micro-benchmarks. Infact, I would call them macro benchmarks. A few server consolidation scenarios are setup and the performance of Xen and OpenVZ is compared. Then, we dug deeper to find out the reasons for the problems using Oprofile. It's not comparing orranges to apples, because both technologies are to
Other VM options.. (Score:3, Informative)
I'm setting up my "next generation" home linux server, and looking into the virtualization options for that. Probably a bigger factor than performance is the setup and manageability. I have found Xen to be pretty primitive compared to VMWare.. setup is a pain, documentation is spotty, and support is minimal. The one advantage of Xen is that you can (and often must) do everything with it from the command-line. The GUI tools are weak at best.
I am now leaning towards using VMWare server. But, I still need to do some testing with KVM.. articles I have read about it sound very impressive. KVM paravirtualization performance is supposed to be excellent. But, I don't know about management.
Re: (Score:2)
That's what I thought, too. Until I installed CentOS [centos.org] (*the* RedHat Advanced Server clone). Start virt-manager, click Create, click next-next-finish and voila, you have a window showing an installation.
Lousy virtualization, Happy users... (Score:3, Informative)
Lousy virtualization, Happy users: FreeBSD's jail(2) facility [ukuug.org]
Source: UKUUG [ukuug.org]
Tags: ukuug, presentation, freebsd, jails, poul-henning kamp
Slides (2.7 Mb) [ukuug.org]
Lousy virtualization, Happy users: FreeBSD's jail(2) facility by Poul-Henning Kamp (phk@FreeBSD.org)
It's not always about performance (Score:5, Informative)
Note that I'm not a Xen apologist, I'm not denying a performance hit here.
Appples and oranges... (Score:4, Interesting)
Xen, is like VmWare or Qemu and provides an independent virtual machine for each system. These systems can be anything at all: Windows, Linux, BSD, whatever.
Performance wise, OpenVZ is bound to win, because it is a different solution to a different problem.
isolation and ease of use (Score:1)
As long as the performance is not bad, I'll take isolation and ease of use over an edge in performance any day. If performance is not good enough, I'll buy a faster server. If the virtuals start disturbing each other, you won't be able to fix it easily and it might be very expensive on the long run.
I have to admit I don't have a lot of experience with Xen, and even less with OpenVZ. If I was building a server at home I would probably use Xen as it's free. At work I'm happy with VMWare ESX 3. Might be the
Re: (Score:2)
Re: (Score:2, Informative)
In fact, free OpenVZ has most of what you tell about VMware. Live migration is here, and it does not require to have a SAN or a dedicated NIC, or pay for vmotion. High availability with clustering can be set up (see here). And finally, you don't have to pay a performance penalty if you want virtualization.
The commercial product based on OpenVZ (i.e. Virtuozzo) has all that plus web-based and gui management tools, P2V migration tools etc. etc.
It also makes sense to point out that Xen also has a live migr
Hypervisor for both (Score:2, Interesting)
Other way around for me (Score:1)
So maybe OpenVZ has some improvements over the commercial variant (seems backwards) or the article is talking about an old Xen2?
I'm currently moving onto a real hardware colo system, which is more hassle, but gives me more control.
Re: (Score:1)
This probably relates to the fact that people usually run 5-20 Xen guests and 50-200 OpenVZ VEs on the same box.
In OpenVZ, you should not take a look at what free or cat /proc/meminfo shows you -- instead you have to take a look into /proc/user_beancounters to see how much RAM etc. you have.
Much better isolation and security? Hmm... (Score:3, Interesting)
I guess I have to correct you here. Xen trades off performance for an ability to run different kernels, and this has nothing to do with either isolation or security. So, Xen is good when you want to run different kernels (different OSs).
OpenVZ, on the other hand, employs a single kernel model, which makes it suitable for you if you only want to run Linux (different distros are possible, different kernels are not). But in this very field OpenVZ is way better than Xen -- not only in terms of performance, but also scalability, manageability, density, and usability.
Speaking of isolation and security, OpenVZ runs on thousands of ISP/HSP servers, and everyone can buy a VE (Virtual Environment) for about 10-15 bucks a month. There one have a root account and can try to exploit the system in all the possible ways. So far those HSPs are not out of business yet, that practically proves the system is secure and properly isolated. More to say, security comes from the constant care, and we (OpenVZ team) do care for security a lot, see this blog entry [openvz.org] for some more details.
Re: (Score:1)
Re: (Score:1)
Re: (Score:1)
Re: (Score:1)
Pradeep