The issue with stopping parents rights the second a child's health is put at risk, is that it invites over officious idiocy from child services, like "oh my god, I saw some snot dribbling from their nose once, therefore you're not cleaning them regularly enough, and their health is at risk!"
Fortunately, not everything has to be a slippery slope.