You do realise that proximity and exposure is the biggest factor in determining who develops antigens to any given disease?
Um, yes? But the diseases we vaccinate against aren't 100% lethal or 100% sterilizing.
It's not chance that determines whether someone who does catch the disease will survive as a reproductive individual. It's the overall strength of the immune system and fitness of the individual.
As long as some catch and survive a disease, evolution selects for the genes those individuals have versus those that die, become sterile or never catch the disease. Take away the risk of catching the disease, and those genes no longer have an advantage. With vaccination, those with weaker immune systems have an increased chance of surviving until reproduction, and as a result, the next generation will, on average, have weaker immune systems than if the culling had taken place.
If your father would have died from measles as a child had he caught it, due to him having a weak immune system, and he survived because he or those around him got vaccinated, chances are higher for you to have a weak immune system than the child of someone from an area without vaccinations. And if you have a weaker immune system, the risk of allergies is higher.
Of course, your father might have had a strong immune system and laughed off measles. But the reason we do vaccinate is that not everybody does. There will be lives saved, or we wouldn't do it. Even if just some survive that otherwise wouldn't have, this will have an impact on the next generation.
We choose to save lives now, and accept the genetic costs of the weakest not being culled from the herd. This isn't something that is disputed. It's a moral choice we make, but we don't get to escape paying the price - at least not until we reliably can make genetic repairs.