The argument against PCIe is that if anything on that particular server fries, all of your data is now trapped inside a dead server; there's no way to fail over quickly. If you care about high availability you probably want to look into a SAN flash appliance, i.e. Pure Storage. (shameless plug)

This is generally true; shared _disk_ will suck once you have multiple VMs accessing it because they'll each try to optimize for seeks internally, but the combined I/O stream will be relatively seek-heavy and wreck performance. Where I work we call this the "I/O blender."

Solid state storage can solve this, but then you have to balance cost, high-availability (if your data is on a direct-attached PCIe card, what do you do when that motherboard dies?) and performance.

This is the problem that a new generation of storage companies are trying to solve. Shameless plug: Pure Storage. We make a solid state storage appliance that uses compression and deduplication to bring down the price of flash while still beating the pants off performance disk arrays.