Well, the fuzziness of patents has something to do with that. It's easy to justifiably think your patent is infringed and be wrong.
Patent trolls, on the other hand... yeah, they should pay. Not only pay, but pay triple as punishment. Patent trolls are generally the exception... they just make the news a lot more often.
If you're a small company that comes up with something nice, patents it, shows it at a conference... then see Competitor's product (who was at the conference too) suddenly have the core of your invention in their product next year... you're going to rightfully sue for infringement.
Maybe Competitor looked at your patent, realized there was a way to work around it, and went ahead. This happens ALL THE TIME. From the outside, very possibly without knowing what they did to get around your patent, it looks like your patent was infringed.
Competitor, knowing they'll almost certainly win and be paid back... could rack up huge legal bills in the process of crushing you.