It is difficult to get PCI passthrough get working with modern hardware - especially using libvirt.
I upgraded my main rig to a 6-core AMD about a week ago with two video cards purely to run Windows in a virtualized environment with its own dedicated passthrough card. I wanted to stop dual booting and run Ubuntu as my main OS, as it's also my main OS on my work machine. I specifically bought a chip and board that support IOMMU for exactly this reason. (Not that I'm complaining about the upgrade, it's a big jump from the older Core 2-based chip and board I had before.) The goal was to eliminate the secondary workstation on my desk, and use one (beefy) machine for both gaming and everyday productivity such as coding/compiling, watching movies, SSH out to various places, etc.
A week later, having patched both the latest libvirt and KVM sources, I still cannot get either the Radeon HD5870 or the Nvidia GT520 to pass through to a VM. Part of the problem is that most (all?) modern cards have more than one PCI function on the bus (VGA adapter, and usually an HDMI audio device) which makes libvirt very unhappy - it won't (without patches) let you detach the device from the host OS when there are devices sharing the bus. I patched around this issue eventually.
The next problem I encountered is that if a device employs GART (so, basically, every video card since AGP 1.0), it's unlikely to work with KVM. I also can't get past the fact that KVM refuses to enable the passthrough device if it shares an IRQ with anything. Really? In 2012, on a *brand* *new* motherboard, I have to start worrying about IRQ assignment conflicts? My board unfortunately does not have any options in the (mildly irritating EFI) BIOS setup for reassigning IRQs manually, either. Damn, looks like KVM is a no-go.
I even tried a very recent version of VirtualBox, as it supports PCI passthrough as well, but GART again rears its ugly head, and VMs with graphics cards attached lock up fairly early in the boot process - my guess is when the guest OS tries to init the graphics card's PCI resources.
VMware supports PCI passthrough for devices such as RAID and network cards - IF you run a fairly recent ESX hypervisor on the bare metal. Which would mean that my main workstation AND the gaming rig would be VMs, so I'd have to keep a separate box around just to run the vSphere client to access my machine. I didn't even bother going down this route, as there was no mention in any of the docs I could get my hands on.
It may work with a very specific combination of hardware and software, but you'd better do a shitload of research before diving into the PCI passthrough world. I ended up sticking with Windows 7 on the box simply so I didn't have to give up my various Windows-only games, and keeping my secondary workstation around so it could run Ubuntu.
HTH.