Yeah, its definitely a children's show. that's why its broadcast on pre prime-time slot on holidays, and then on Saturdays in the 5-6pm slot. That's typically before the sports start. Incidentally, its the same time-slot that the A-Team was broadcast, so you can kind of see the target audience age (and the reason why there's always some form of totty sidekick present for the Dads who have to watch it with their kids)
However, I think the 'modern' doctor series have done remarkably well in appealing to an older audience as well.
That's part of the viewing differences between the UK and the US. The A-Team was always televised in the 8pm time slot back on NBC, not 5pm or 6pm as in the UK.
As others have stated, plenty of us US fans grew up watching Doctor Who on PBS stations that generally televised it beginning at 10pm on Saturdays. That's how I remember it. And since for the most part they focused on the Tom Baker era, it was rather dark and creepy enough to give the impression that it was a show for adults. One of the first episodes I saw - when I was 10 - was "State of Decay" and that was creepy. Hell, "The Keeper of Traken" was creepy too.