Possibly because our "passion" is not driving fast? My personal passion is golf and playing with my kids at the park. In order to get there safely, and get home safely, I drive like a sane person. If your passion is driving fast, then find a safe place to do it. There are tracks everywhere, and some small airports will rent out unused runways as well.
Just because you have a passion for something, doesn't mean you can indulge in it 24 hours a day. I also have a passion for shooting AR15s, but I don't expect to be able to blow off rounds in my neighborhood or in a grocery store parking lot.
What you call 'having fun on the roads', most people would call being a complete jackass.