Filing a law suit should at an absolute minimum require the plaintiff to pay some costs to the defendant (perhaps the smaller of the legal costs incurred by either side) should the plaintiff lose. This will help to minimize the number of frivolous lawsuits and so minimize the quelling impact of such lawsuits on society.
Aside from reclaiming legal costs, how about the plaintiff stands to lose the amount of compensatory and punitive damages they seek? That'll bring frivolous lawsuits to a halt with a quickness, and will probably keep amounts sought reasonable in legitimate suits. (Keep the appeals process in place, of course.)