1) They set a minimum price far too high. Relatively few mods are worth a dollar, even the ones that are worth buying at all.
I agree to a certain extent. For Skyrim certain mods have just become standard fare to install, like SkyUI which makes the UI at least usable. I'd be happy to pay a dollar for SkyUI (ignoring the whole SKSE thing for a second) if that would rid me of the default UI for Skyrim for the 200+ hours I've put into the game.
2) They didn't protect from fraud.
This was in my opinion the worst problem with the whole ordeal. Not just fraud, just the fact that they barely checked what was going on with the mods in question. Even their rules for mods that used other mods made it clear that they really didn't give it too much thought. A lot of mods are frameworks that make developing mods easier, or that make modding possible in ways the API of the creators of the game didn't allow (eg SKSE). With SKSE you start entering this legal murky area and I can't help but feel that Valve never gave it too much thought (and neither did Zenimax/Bethesda).
On top of that, modding communities are rarely good places to build a business in. Most of these people aren't professional developers, and while it's not unheard of that amateur developers can build a sane business model, let's not kid ourselves here. The minecraft modding community is the perfect example of amateurish behavior and so much drama. I don't want to generalize that entire community, since there are a lot of people doing a lot of neat things with that game, but there's been more than a few cases of a modder purposely breaking the game when another mod was installed simply because of some stupid fued between them.
Lastly, there's an implied expectation of a consumer that a modder will maintain his work when a game gets updated if he's paid to do so. Few publishers release preview builds for modders to work on, and even if they do with many of these amateurs even that wouldn't be a guarantee that they would update their mods in a timely fashion. Quite frankly, if I have to pay for SkyUI, I expect it to work without too much problems even years after I bought it.
3) They didn't share the profit well.
I'd like to agree on that point, except in the end Valve just agreed to the terms of the publisher and did the math on their own costs. But let's be honest here, Skyrim as a 4 year old game won't be getting anymore updates. Basically, from a community point of view, that's just Zenimax/Bethesda being greedy at this point. It's like a city council deciding that they're going to charge an admission to the sandbox if you're planning on building a castle, while the whole thing was built with community taxes. (Yes yes, not the most accurate of metaphors, but at least it's not a car) Sure, Bethesda could say something like "Yeah, we were planning on using that revenue to keep the game updated, provide a more complete API, interface with the community" etc etc, but let's face it. Bethesda is working on newer titles, and anything and everything Skyrim is just bookkeeping from now on.
What it is, is perfectly in line with the vision many of these companies have these days about what a community means to them. While I'm not a fan of the so called "Let's play" videos, if you look at the whole drama there about monetizing it's a perfectly good example of what's wrong with many publishers these days. Many publishers want a cut from the video revenues today, while in essence it's really free advertising they're getting. The arguments being made are that most people won't buy the game if they can just watch it being played online, but I don't think that argument really flies. To me, Let's Play videos are kind of like a gameplay video. Before I buy a game I go check it out on a Youtube channel that isn't clearly a marketing channel, and then decide on the gameplay I see if I buy it or not. If your game is so simple that a Let's Play ruins the experience or doesn't entice me to buy it then more than likely it's not a very good game. So let's be fair here, and say that if you are basically taking a cut from someone who's making free advertisement for you, that you're operating on the premise of milking your community instead of building one.
4) They launched it suddenly, with no notice. Nobody had any inkling it was coming, least of all the modders who would be most affected by it.
Their biggest mistake in my opinion was doing this with an established modding community, especially one the size of Skyrim. The move was bound to be controversial, whatever game they picked, but they should've just gone with a newer title with a less established community to test the waters. Skyrim as a testing bed was bound to fail as for the existing userbase it would be too bitter a pill to swallow.
Gabe Newell made the mistake of going to Reddit to make some quick one-liners as a way of dealing with the shitstorm in his mailbox. While he did make a couple of good points there (especially on the topic of accounts getting banned on Steam for criticizing the paid-mod change being a really bad idea), his quick one-line replies offered little to no reasoning with the community. He might as well just not have done the AMA at all, because let's face it, all you're doing is painting a big target on your back in this kind of situation. Kudos to him as a CEO to try and engage with the angry mob, though, as I doubt few other CEOs would be willing to do that.
As a final remark, I think Valve is a really oddly run company given its revenue. For a company that seems to more or less know what its consumers want and have both experience as a developer, publisher and the most successful distribution platform for games this whole ordeal has made them seem ignorant of the communities they cater to. From my point of view, the stunt they pulled just illustrates how Valve mostly depends on "lucky guesses" for their business. Greenlight is another example of ill-conceived ideas at Valve at trying to make the platform more user-driven. Skyrim may have been the smartest move from a market potential point of view considering the size of that modding community, but the dumbest from a community point of view given the size of that community.
All in all, I think it's a missed opportunity for developers of games and mods. Think of it like this: a steady revenue stream from modding could've enticed many publishers to build better APIs and tools, offer some basic support to the modding community, etc. A developer could've kept their update cycle going a bit longer with the revenue from paid mods if successful. But taking a 4 year old game that won't see anymore updates, slapping "paid mods" on it and generally being disruptive to such a large community is what leads to the situation Valve finds itself in now. The whole concept is now extremely toxic and no dev is going to be willing to touch it for years to come.