Remember censorship starts with the "think of the children" thought pattern.
I can really see both sides of the argument, but I do agree that your point is valid.
If we step away from books for a moment and consider television, the point becomes clearer. On one hand, I would never want to censor my children from watching something about Jesus or hosted by Carl Sagan. Those are interesting topics no matter which side of the debate you may lie on. On the other hand, I wouldn't want them watching a majority of the crap reality shows, extremely biased political talk shows, and other trash that has very little value for a child. If my child is interested in politics, I will encourage unbiased sources (all two of them) to make sure my child isn't getting brainwashed by propaganda.
If we bring the topic back to books, I think you can see some similarities and of course it comes back to "think of the children". I wouldn't want a book to be banned because I disagree with the content. I would want a book to be banned because it is fucking nonsense representing itself as the truth.
To reiterate what I am trying to say, I can't even really think of a book off-hand that would fit this criteria. The closest I can come up with is The Book of Mormon. It represents itself as historical fact when it should clearly be categorized as fiction. Even despite the tales of an angel coming down and speaking of golden tablets and such, I don't think I would necessarily want the book banned; however, I can see both sides and if books should be banned, The Book of Mormon presents a decent case.
I want my children to read what they like to and to not be exposed to indoctrination at school. Unfortunately, with as polarized as our country has become, it is increasingly difficult to make sure some extremist isn't plastering their disingenuous views on students. I know, my point kind of sucks here, but in the end I think saying "banning books is okay" is not a correct statement but I also think "books should never be banned" is false as well. The reality is somewhere in between and is decided upon with rational thought and a lack of bias.