I've never quite understood how you can say "Allowing (x) to happen imposes your viewpoint on me".
It's a good day to extend that thinking.
Allowing (x) to happen does several things:
1. Signals social approval of (x)
2. Increases the frequency of (x) as a result
3. Creates social consequences of (x)
4. Disallows a society where (x) is not present
Let's look at these one by one.
First, your society is signaling to its own member that (x) is not just accepted behavior, but thus is recommended behavior. If we legalize eating raw octopus, we have said nothing is wrong with it; that puts it, in the binary of behaviors described by government, in the "approved" category by not being in the "disapproved category."
Second, that means more people are going to do it
Third, this means that all of us are going to experience the social consequences of it. We are all subsidizing it, in effect, even if we disagree with it.
Fourth, you have eliminated my ability to have the society I want, which doesn't include it.
Let's look at marijuana.
1. We legalize dope. You now have no reason to tell your kids not to do it, since gov't thinks it's OK..
2. People smoke more of it.
3. Whatever social consequences of pot-smoking occur and we all pay for them instead of putting that money toward other things, like space exploration or ocean renewal.
4. I lose the ability to live in a society where pot-smoking is not normal. I may want this for moral reasons, ethical reasons, or even scientific reasons. But either way, I'm deprived.
You've fallen into a fallacy:
How is "You may do this, or may not, depending on your choice," more imposing than "You may not do this"? How in the world is freedom more imposing than restriction?
You're looking at a change in state of the law, not a change in state of society.
Either way, permission or denial, a change has been effected and that changes the overall experience of the society.
Calling it "freedom" (etc) is just a linguistic and political trope in this case, as it doesn't relate to the effect of what you're describing.
Permissiveness is not victimless. It is simply a change in status, much like denial. Thus, any condition is an imposition in effect.
Currently, our society has a bias in favor of permissiveness, using the "it's not a change to you" argument that you outlined above. However, this is fading, since people are seeing that all these permissive changes have long-term social consequences starting with the perception of approval.
Hope that cleared it up for you.