I personally don't understand why a business would turn someone away due to the color of their skin or who they want to bang where. Money is money is money. People who discriminate like that are assholes and deserve to be boycotted.
However, the Supreme Court has ruled that freedom of association is an integral part of the first amendment. You also have the right to believe in flying spaghetti monsters. And I have the right to tell you I think you are dumb for believing in flying spaghetti monsters. If we don't like each other, we are the right to not associate with each other. If you tell me to go away and I show up at your house in the middle of the night, you can have me arrested.
If I prevent you from shopping at my store because I don't like you or your lifestyle, I'm wrong. Isn't it also wrong to force someone to allow people to shop at their store?
Freedom of religion and freedom of expression also means that the KKK gets to parade down the street and have a website. It means that people you strongly disagree with and believe are morally wrong have the right to run shops and churches and live their lives. It means gay people can walk down the street hand in hand and bakers can tell them they are living in sin and going to hell. It means you can stand in front of the bakery waving a rainbow flag. If you remember store owners telling people their skin was the wrong color, you also remember when it was unpopular to say non-whites deserved equal rights. But we let those unpopular views be expressed and got the civil rights movement. We let gay rights activists speak out against the majority.
If you want bakers to bake cakes for gay weddings, do you also want bakers to bake cakes for the KKK? Who gets to decide which cakes MUST be baked and which are reasonable to refuse? A Walmart employee refused to bake a cake for a cop's retirement party a few months ago because cop cakes are apparently racist. Should that employee go to jail? The Walmart fined?
There is a difference between criminalizing, fining, and forcing conformity and criticizing a view you find reprehensible.