The Fifth Amendment only addresses the competency problem as it applies to one type of evidence, and still leaves courts dealing incompetently with every other type of evidence.
So since the 5th doesn't solve every problem, only one, then it has no value and should be thrown out?
If judges are really so incompetent that they don't understand how someone can become flustered...
Let me stop you right there. It could be argued that judges are, as a group, above average intelligence (*citation needed) and would understand this. The problem is that we all get trials by a jury of our idiot peers, who are by definition at least 50% below average intelligence. If we're tried by an bunch of intelligent, competent, fair persons then sure, make 'em talk. The problem is that we're being tried by a bunch of common people ("A person is smart; people are dumb, panicky, dangerous animals, and you know it.") in circuses presided over by the most reviled members of our society (lawyers) and the ones of that cabal who fought their way to the top (judges).
In that environment, I'd like the right to keep my mouth shut, thank you very much.