People who are ugly, inept, and poor are not welcome to join the "real" greek system.
This may have been true in the past, but it is not anymore, at least not universally. I grew up on a farm in the middle of Iowa, not in poverty but certainly not well-off, and when I went to college, I was recruited by an old family friend to a fraternity. I was very uncertain at first, but the rush chairman was a nice guy (he ended up my "big brother" and I followed in his footsteps as a materials engineering major). I came down and stayed in the house for orientation, and after a good look at the dorms I'd be living in, I joined the house.
Honestly, living in house is cheaper than dorms, we've got several people who came in very socially inept (myself included), and I'm not a great judge of male physical appearance, but I know there's no looks category we factor in when deciding whether to rush or activate someone. In fact, the only problems we've had activating people dealt with more of the "scumbags and assholes" mentioned in an uncle post. There are several houses on our campus, chapters of our fraternity on other campus, and even our house years ago that seem to exemplify these qualities; we use these as examples what not do when we plan out what we want the fraternity to be like.
The Greek system as a whole is undergoing a lot of changes now that a lot more options for incoming students are opening up. Our chapter is one of very few on our campus that is actually growing right now, and I'd like to think it's because we don't treat people like shit.
And before I forget, as to the grandparent's point, I do think it was very beneficial in my case to live in the fraternity. I'm the type that doesn't make friends as easily, but once I do, I make good friends, and the fraternity was very conducive to this. It's also provided activities that help me branch out more. I can easily picture myself cooped up in a dorm room for four years, but thankfully that didn't happen.
The bottom line is that if people in fraternities you know do act like complete dicks, they probably are and you should steer clear of them. But if they do seem like nice guys, it could be because they are. Even little stereotypes like the ones people hold against Greeks can be damaging, so try to get to know them a little and then make a judgment. And if you have done this and happened to only run into asshats, I am sorry.