The biggest down side for me, as I understand it, isn't so much having different pricing schemes for different traffic. It's the ability to re-direct or block traffic the ISP doesn't like. Let's say that
Time Warner and Yahoo! enter a deal. Suddenly people who have Time Warner as an ISP can't go to Google. Or any other "competing" web-site. Or even if you can get to the competition site, it's 5x-10x or more slower than the preferred engine.
Carried out to extremes, it could mean that you could only visit Turner broadcasting (subsidiary of Time-Warner) sites in a reasonable manner, because they're the preferred sites. And they could block any content that might be seen as infringing on any of their corporate copyrights, regardless of whether it does or not (no DMCA counter-notices when the DMCA wasn't involved in blocking the material).
At its worst, it gives the ISPs the ultimate control over the content their users see, and how they see it. Would it get that bad? Honestly, I don't think so. But knowing that nowadays a lot of people get service from a single provider (cable, Internet, phone), a significant number of people might find the idea of switching providers tougher because of all the other changes...if they even have another option (besides dial-up or doing without).