In pretty much every HPC cluster I've seen or been personally involved with (mostly oil/seismic processing or crash simulations), the type of CPU is only one of the cost drivers!
Typically you end up spending about as much on fast interconnects as you do on motherboards/cpus/ram etc. The main exception to this rule is when you have an embarrassingly parallelizable workload, with small memory footprint and no need for cross-system communication, i.e. like a Monte Carlo simulation or password cracking.
For oil we used the largest single-image NUMA/SMP machine we could get at the time, this machine did the initial gridding of the problem space, then a standard cluster of 1K dual-cpu motherboards (i.e. 2K cpus) took over and did the main part of the actual processing.
There are exceptions though, like if you are doing Linear Programming type optimization which can be really hard to parallelize, or if you are using very expensive SW:
When you pay more for the SW than for the HW it is running on, then it makes sense to use bleeding-edge (gamer type) cpus.