As a long time VisualStudio user I find your problems to be rather 'bizarre' to say the least. I use many IDEs on a day to day basis from XCode, Eclipse, MonoDevelop, Emacs, KDevelop (only for one old project), and VisualStudio 2008, 2010, and 2012. It depends upon what platforms I'm building for.
I have a rather large Visual Studio solution that contains more than 30 projects including web services, DLLs, controls, assemblies, client applications, and COM/DCOM objects, and it takes about 5 seconds to startup and be ready to work when I load that solution.
I also don't run into the incremental build issues that you seem to experience. I've been building huge projects in VisualStudio for more than a decade and never experience the problems you are reporting except when somebody was hosing up source control and VisualStudio though files where changing all the time.
That being said, XCode is a really nice environment as well.