No, benchmarks do not become a double-edged sword. They are simply misused and misunderstood.
An automated software benchmark has value only to a developer or development team evaluating their own code. Anyone using such a benchmark for any other purpose (like comparing the performance of their code to the performance of others' code) is misusing the tool, and in doing so potentially drawing faulty conclusions.
Automated benchmarking tools are not a perfect measurement and they never will be. They are not an and-all solution to anything. They are simply one way to do a quick (and possibly very dirty) analysis of your code. To be useful at all they must be properly understood as such.
The real issue here is this: too many people either believe or like to pretend that some particular benchmark is a god. It isn't, and their naivete or pretension is leading them and others astray.