Increase the patent filing fee by some flat amount plus some amount that's based on how many patents the filer has had accepted and has has rejected in the past year or two (this curve should be fairly steep.) Put a portion or all of that increase in escrow. Once patents are published for public review, if someone reports to the patent office prior art that causes the patent to be rejected, the first person to report that prior art receive the bounty. [In the case of multiple pieces of prior art causing the rejection, split the bounty between the first submitters either equally or based on "how compelling" the art was to the rejection.] If it's rejected for other reasons (the filer's fault for trying to patent something obvious, for instance) the patent office keeps the bounty.
If the patent is accepted, either: 1) return the increased amount to the filer, 2) give that extra amount to the patent office [I don't really support this since it could lead to greatly increased pressure to approve a patent for the kitchen sink, not that that's not a problem now], or 3) keep it in escrow (and making interest for the patent office) for a period of time in case someone successfully challenges with prior art in the near future, at which point they receive the bounty.
People who file one patent won't suffer more than the initial flat increase. Companies that file many patents that are all accepted won't suffer more than N times the flat increase. Companies that use a shotgun approach to patent filing will eventually see the "points" on their "patent license" increase to the point where it's not financially viable for them to file more; they have to wait until some of those failed filings age and come off their record.