I've never personally worried about people taking pictures of my kids but I'm a little self-conscious when I bring my DSLR to the park to take pictures of them playing for the grandparents -- like, if there are other kids there with their parents I try to not use the camera. When the MPs roll by on their routine patrols I always half-wonder if they're going to stop and ask me (an almost-middle-aged male) what I'm doing with kids and a camera.
I'm not sure where my caution comes from, I think it may be just from stories I've read about like this. I can see ubiquitous cell phones not being an issue, just because half the time when you see someone with a phone out they're taking a picture, but I always feel a bit weird with the full-sized camera, like there are unseen eyes judging.