But you missed a fourth group, which is the people who enjoy your novel and would have been happy to pay for it if they'd found a legitimate source first, but who actually found it somewhere else and maybe even paid the unauthorised source for it instead of you.
As someone else doing relatively small-scale creative business, I can testify that this group can be a significant one, and I both sympathise and empathise with the situation that sneakyimp has described in terms of watching people ripping your stuff on YouTube and finding them hiding behind the safe harbour provisions even when it is clearly an ongoing problem and you have made them well aware of it.
I'll add a little from personal experience. Under the DMCA and similar laws, safe harbour is rarely an absolute protection, and typically those appealing to it are still expected to have some mechanism for dealing with persistent infringement, such as stopping the account of the infringer. In our experience, YouTube have shown no willingness to do this, and any attempt to contact them about it at their published email address for such matters just gets something boilerplatey back that directs you to their online form for filing a single DMCA complaint... again. Perhaps when they're dealing with anyone big enough that they are likely to take real action and have the resources to do it, YouTube follow other processes, but from a legal point of view it looks like they would have forfeited their safe harbour protection in a case like ours if we'd wanted to make a point of it.