You don't have a right to my property.
I have to laugh at this. Of course he does, through force of law. You essentially rent the land through property taxes, and you have to abide by zoning rules and ordinances. You have to meet certain standards with your construction, and the house must be serviced by certain kinds of utilities to be rated habitable. And if you want to enter the commercial sphere you have to follow the rules of the market.
None of what I said is even remotely controversial or new, and if you disagree with that it is you who are the special snowflake.