Second, I don't think that people are out to screw me personally. At least most people that is. But I do believe that humans take the path of least resistance.
Bingo. I am not part of the crowd that downloads music/books I haven't paid for (unless offered by their creator for free), yet I understand that business models in this realm must take human nature into account when setting pricing. You have displayed an understanding of applicable human nature.
Simply put, you need to find a balance where people who value your book enough to pay for it can find it at a price they find reasonable. If you price the book too high, you will lose sales/income to piracy. On the other end if you price it at the cost of publication very few people will pirate, but you'll make nothing. The aim, therefore, is to set pricing where the product of volume times gross margin is highest.
Also, don't be fooled into thinking that intellectual value or degree/amount of effort translates directly into economic value of the work (as individual copies or when multiplying price times volume). Because your book is great doesn't mean it is worth 5x the price of a novel. That your market is smaller doesn't make it worth more per copy either. Economic value is literally that - what people are willing to pay. I find it absurd that autobiographies by former presidents have higher economic value than books like yours, but that is, well, the ugly truth.
Lastly, on suing people. Even on Slashdot I think you'll find most people would be supportive of suing businesses or individuals who are making money by selling illegitimate copies. Such a commercial benefit puts them squarely in the wrong legally and morally. To sue them or not depends on to what degree you're willing to sue them out of principle (to deter this practice) and how much return you might get from such.
On the other hand, suing your "customers" is a very bad idea. Even if you can extort some $$ from them, I would avoid suing people who are interested in your work but have obtained it via other means because, to them, they found it more reasonable to do something which is likely immoral and may have taken more effort, than pay the price you set.