I simply have the right to decide what I put into my body, regardless of what you think about it.
If you had the magical ability to not spread viruses, I would accept your statement, but you do not. When you have measles, it is exactly the same as walking around town with a handful of hypodermic needles, injecting random passers-by with viruses. You violate everyone else's "right to decide what they put in their bodies."
You live in a society, made of other people. Therefore, you have some responsibility towards the others who you interact with to provide you with food, gasoline, clothing, education, fire protection, etc., etc., etc.
The good news is that vaccines don't need to hit 100% of the population to be effective enough to prevent an outbreak. "Herd immunity" prevents the wide spread of a disease when most of your neighbors are immune. A level somewhere between 80-95% vaccinated is enough to stop an outbreak. But that's a very high level to achieve voluntarily. Vaccines are ineffective in some people. Some people with auto-immune diseases, or undergoing certain therapies, or are just too frail, can't risk taking some vaccines. And some people are so isolated by either geography, finances, or intelligence that they lack the opportunities to learn that they need vaccinations. Between those groups, there is almost no extra safety margin for tolerating people who think they deserve some special exemption because they "believe in" something divine, or think they have some special rights that they themselves violate on a daily basis.
We don't have a special "isolation island" to keep unvaccinated people from putting the rest of us at risk. Instead, we pass laws that enforce schoolchildren to put something in their bodies, or else we deny them schooling. But that's all the control we have, so far. Instead, we have to rely on public health education, and hope people voluntarily comply.
What we really could use would be swift punishment for the anti-vax deniers. Unfortunately, that crosses swords with free speech. So instead, we have to hope we can convince people that anti-vaxxers are stupid, hostile, anti-social jihadist monsters who are trying to destroy humanity with their lies and bioterroristic weapons. It turns out that a disturbingly high number of people are so extremely gullible or stupid that it's not as effective a strategy as we need.