It's one thing to have some Larry Wall style eccentricities, but Stallman hurts any movement he attaches his name to because of his extremist views. He believes, for example, that programmers should not expect to be paid for their work and that it's more important that non-free software disappear than it is for someone's children to be fed (he also believes nobody should have children). He's also made vile statements about what he calls "voluntary pedophilia", claiming that it should be legalized.
The annoying part is that in nearly every Stallman discussion, people will say things like, "You may not agree with everything he says, but we sure need someone like him who always sticks to their guns!" No, we don't. He's hurting the movement.
GNU was an interesting philosophy when it was started, but it's not as if it was the only open source ideology or that other open source movements wouldn't have taken hold. This isn't to diminish GNU so much as it is to diminish Stallman's glorified role in history among computer geeks and lessen the movement's reliance on a crazy person.