I routinely run 7-8 VBox guests with decent performance on a very modest dual socket dual core 8GB Opteron box - but under some version of Solaris, at this moment OpenIndiana 151a. I keep VirtualBox updated to the latest version (I'll put 4.1.4 tomorrow, though). I use it for network testing - Solaris, CrossBow and VirtualBox make for a very decent lab environment, not to mention ZFS snapshots and DTrace. The guests include three W2K8R2 DCs, some OpenBSD firewalls, mix of NetBSD VMs, two used to build -current and pkgsrc all the time, some W7 as well. Memory may be tight at times, but still it manages.
I've never ran VirtualBox hosted by Linux and do not intend to even try it (actually, I don't have a physical machine under Linux at this time). I do run a few Linux guests here and there, on my laptop while it is under W7-64 for example - Bodhi is quite nice. They usually cause the most problems - getting the right kernel source, development environment, headers and the rest.