I guess the only question is, how far do you take it when determining that somebody is harming their children. I definitely think that everybody except the tiny percentage of individuals who have a medical condition should be vaccinated against things like measles. But I'm not so sure about things like chicken pox or the flu vaccine.
If your child is stricken with a disease that has high potential of death, significant injury, loss of function or disfigurement or the same to others, and that disease is preventable through vacination, and you failed to provide that vacination that would meet the standard of harming their children or endangering the public in my mind.
As far as flu, people die from that, my Mother was hospitalized just last month for the flu, while my Dad was in the ICU after arresting while being treat for pneumonia that was as likely as not to have been triggered by having the flu. My Dad never came out of the hospital and was on a respirator for 6 weeks; Mom back in the Hospital because she never regained enough strength and is now refusing treatment so she'll pass away soon too.
My attitude right now is to tell the antivaxers to STFU and get the Kids their shots, if your kid goes deaf because of a fever due to having measles, I'd throw your ass in prison.