As someone who touch-types Dvorak at home, and has to switch back to QWERTY at work, I think I can safely say my experience trumps your few symbol keys moving around...
I'd argue that no, it actually doesn't trump it.
IME it is *far* easier to switch between two completely different systems, than to switch between two systems which are exactly the same, except for one or two minor parameters.
Consider a Brit, who fluently speak both English and Russian, conversing with two people; one of whom speaks Russian, and only Russian; the other speaks US English, and only US English. When speaking with the Russian, the Brit's brain need switch to and maintain Russian only once. When speaking to the USian, the Brit can speak in their native tongue - except when certain words come up, which the brain must anticipate, and engage to translate those to US English.