It does not hurt you for someone to believe that the world was created by a flying spaghetti monster or aliens or green mold. It does not hurt anyone that people want to believe that invisible space monkeys have a plan for them that involves them giving food to the poor. It does not hurt anyone if someone wants to believe that the world magically sprung into being cause their invisible magic man cried or something.
While those specific examples are apparently harmless in themselves, if the belief construct someone uses as their guide urges them to serve as the mobile deployment device for a significantly energetic chemical reaction (as an example), I would argue that their belief system had a direct impact on my well being.
In a similar vein, if a belief system urges an individual to goodness (charity, empathy, etc) as well as evil (restriction to medicine, equal treatment, etc), I think it would be inappropriate in the extreme to "accept the good with the bad" and remain silent out of respect; these qualities need to be publicly separated, the greatness lauded and the evil stamped out.