So then murder is OK so long as I can successfully get away with it. Got it.
How about throwing someone into a jail cell? What about throwing someone in a jail cell because they said something objectionable?
Because what you're saying is, so long as no one opposes me, that's totally cool. Uh huh.
Who says what is legitimate? The "people", are you serious?
What about an island with one person? It's impossible for anyone's rights to be violated, there's nobody else to violate them!
What about an island with two people? Can one kill the other?
What about a totalitarian government ran by a tiny minority? They don't have "the People's will".
What about a court that strikes down (refuses to enforce) an overwhelmingly popular law in a democracy?
If all mankind minus one, were of one opinion, and only one person were of the contrary opinion, mankind would be no more justified in silencing that one person, than he, if he had the power, would be justified in silencing mankind.
People tend not to question the laws of gravity, because when they do, they tend to end up mangled. The same thing happens for questioning the basic laws of economics. If you have no objective mechanism for deciding who is right to act on what, the outcome isn't generally very pleasant.