Using Visual C++ would require windows. The disadvantage of using windows over *nix are many: Licensing, Manageability, ad infinitum.
Our main cluster is CentOS-based using ROCKS cluster software and is running on 208 blades using several different generations of HP BL46[05] G[1-6] blades that get refreshed periodically. We have a mixture of CPUs in production currently, dual dual-core Xeon, dual quad-core Xeon and Opteron and some dual quad-core Nehalem. We're replacing the dual dual-core blades with dual-hex shortly to gain processing power without a power/cool/weight penalty.
We usually disable hyperthreading since we prefer each processor core be used for one job and only one job. Time-slicing between two jobs means that time's wasted flipping back and forth and resources will be used up needlessly. We also size system RAM so there is at least 3GB/core available for each job submitted. Using those strategies, we avoid swap as much as possible and get the maximum CPU utilization. Typically, each core is running at 100% for days on end.
A PS3 cluster would be similar in use, but limited to jobs that didn't need a lot of RAM or disk I/O. Since a PS3 cluster is pretty much a roll-it-yourself endeavor, I don't expect it to get used nearly as much as the production general purpose cluster. As time passes, it begins to look more and more like a toy compared to the newest advances in HPC.
When the PS3 cluster project started, it was one of the "sexiest" ways to get your hands on a lot of power for not a lot of money. Recent developments in GPU-based computing is beginning to attract attention, but not every system is a viable host for a production grade GPU board. System form factor and power requirements are significant issues to overcome. CUDA/Tesla boards show promise, but are only suited for certain applications, they would never be intended to replace general-purpose cluster computing. Typically, we'd want something that can run 24/7/365 with a very high MTBF.