I thought the "dying in the fields" phrasing was an exaggeration you used for dramatic effect, but apparently it didn't originate with you. The argument I was talking about, which is a fairly standard libertarian one (and there's a good chance you've heard it anyway; I just want to avoid possible misunderstandings), runs roughly as follows:
1. Individual situations and lives are complex. Whether a voluntary exchange of goods is a net positive for someone depends on a lot of factors, many of which are much easier to evaluate for the individual involved than for any outside observer. The individual in question also typically has better incentives for getting the answer right than a third party, which (among other things) helps safeguard against many kinds of irrationality.
2. Therefore, not hurting people directly through commercial transactions is usually quite simple: Don't apply force, and be honest. Obviously forcing people into transactions is bad; robbing or raping people is clearly immoral. The same goes for defrauding them in voluntary transactions (by lying about your product, paying them with counterfeit money, or whatever).
3. Aside from those two obvious things, you're usually best off focusing on whether a given possible deal is good for you. If everyone does that, you get the people with the best information judging the efficiency of transactions, and most of such transactions will then end up mutually beneficial.
4. For similar reasons, judging that a given deal between two third party entities is directly morally bad requires that at least one of them is applying force, engaging in fraud, or catastrophically irrational (and also catastrophically more irrational than the actor making the judgement; unfortunately, it's very common for people to erroneously settle on this explanation). This is the standard argument as to why corporations running sweatshops (assuming the work contracts are voluntary, they don't inordinately pollute the environment, etc.) aren't necessarily doing anything immoral, and AFAIK it is valid.
Now, this is just about the direct moral consequences on the actors involved. By engaging in honest voluntary transactions, you might still be doing indirect harm by supporting an evil business. If you buy stuff from Sony, you're indirectly supporting DRM. If you buy NetApp devices, you're indirectly supporting software patent litigation. And if you do business with slavetraders, you're indirectly supporting slavery.
If you were making this last point, then fair enough, but note that this really only applies where you expect your money to go to an evil organization - when you're dealing with someone who's just voluntarily (i.e., not being coerced through threat of violence by other individuals or similar) offering sexual services for money, there's nothing morally wrong about doing business with them.
This also doesn't necessarily conflict with libertarian ideology very much. Libertarians don't typically condone forcing people into transactions, and many would agree that there's a role for government in preventing literal slavery (but note that "wage slavery" is really a horrible misnomer, and I'm not including that concept here).