I think part of the issue is price-point. In the days before DLC, a lot of games came with later "expansion packs" which were essentially a few new levels and some added weapons/units/etc, but essentially the same core experience. Those generally cost less than the initial game, but were still a decent price and often *very* popular (especially Blizzard games such as Brood Wars, Frozen Throne, etc). They weren't much more than mods, albeit by the original publisher.
There were also the unoffical mods such as the aforementioned Desert Combat which - if they made any money - was by donation.
So where does that leave the 3rd-party paid-mod arena. Well, if it requires the original game, then *smart* publishers would put together the fact that "hey, this popular mod is driving extra sales of our game, great!" Some may be want a cut, which may also be something that needs to be worked out. But so long as the price is reasonable, I don't see any reason why a well-done mod would be less popular than an "expansion pack." I probably wouldn't pay $50 for most mods, but $5-20 is still enough to give some devs enough extra cash to make it worth investing the time into making a really polished mod.