There's also long studies made to make sure the streets are safe with the given expected traffic, choke points, traffic, and the fire department has to check it out to make sure it's okay in case of an emergency. It's not an exact science, but shit like this can seriously fuck things up.
Add that a lot of buildings and roads are built on pure corruption (things that really should not have been built gets built on "special" permits that skip normal rules, etc), and you end up with no one being happy.
Then you have people in cities like SF and Boston bitching about skyrocketing rent and too few units being built. Why do you think that is? You're right: they can't stop you from using the roads. So they just vote to stop people from building at all instead. Then the rich get skyscrappers up and live at the top, away from the "peasants". And you increase the gap between rich and poor.
This type of bullshit attitude is why everyone is so stressed, nobody's happy, and we have such a crappy society.
Oh, and while people go "I have the right to use this street! My taxes paid for it!", they'll conveniently ignore the other laws they don't like, speeding, city noise rules, etc.
In my particular case, the street in front of my house is actually a private way, with a compromise done with the city long before my time to give an easement right to connect 2 other street for locals. There's a bunch of (legally enforceable signs, since we own that street) signs up, as you're not allowed trucks, passthrough traffic, etc. People will happily ignore those, AND the street is on Waze to boot.
If it went both ways and all rules were observed, there wouldn't be any problem.