Paying People to Argue With You

Bennett Haselton has written in with an essay on a strange experiment on-line. He starts When you first hear about Amazon.com's "Mechanical Turk" service, which allows "requesters" to pay "Turk workers" a few pennies to complete some task which is hard to automate but easy for humans, what's the first application that comes to your mind? The system has been discussed previously on Slashdot, but I'll bet a week's wages for a Mechanical Turk worker ($1.45, according to one of them) that I was the first person who used it to pay people to write rebuttals to one of my arguments. Keep reading unless you want to fight about it.

The interesting result was that some of the rebuttals were quite insightful, and resulted in me making changes to the argument that I would make if I had to present it again. Judging by the literacy and intelligence of some of the respondents, most of them probably wouldn't need Mechanical Turk as a source of income, so I assume most of them fit the profile of this Salon.com writer and are doing it just for fun. Hell, you can find enough people on UseNet and Slashdot who will argue with you just for free.

But there were a few reasons I found this preferable to the conventional ways of gathering interesting rebuttals to your own reasoning. If you send out a sample argument to all of your e-mail buddies, you will probably get some useful replies, but they may start to think you're a little weird for asking them to evaluate your thought processes, especially if you do it over and over. Post an opinion on UseNet or Slashdot, and you may have to wade through a lot of crap to find the useful responses (while others may consider your post to be part of the crap that they have to wade through). And in both cases, there's the potential embarrassment of what you're asking for -- the risk of seeming so uncertain about your own opinions that you want other people to check your work for you. (I actually think that being uncertain about your own beliefs is a virtue, but it doesn't seem to be one that our culture prizes very highly.) Using Mechanical Turk addresses most of these problems; even though you're still admitting to total strangers that you might be wrong and asking them to shoot you down if they can, at least the evidence of your insecurity won't turn up when your next employer or Internet date does a Google search for your name. ("Damn it, I want a man who doesn't question his bumper stickers!")

So, while I didn't find it useful enough that I would run every opinion through the Mechanical Turk machinery to see what feedback I could get from it (I'm not paying a bunch of them to proofread this article), I did like enough to recommend it to people for certain arguments in certain settings. The main kinds of arguments that I would try out on the Mechanical Turk service would be about abstract philosophical or moral questions on issues that have been around forever, like abortion or the death penalty -- topics so explosive that you'd risk making your friends very uncomfortable if you test-marketed your arguments on them, and which would seem almost rude to post about in a public forum because the debate topics have been around for so very, very long. But on Mechanical Turk, $1 is apparently enough to get people to ignore the awkwardness and the exhaustedness of the topic and to focus on what you ask.

And what was the argument that I used to test it out? Perhaps the geek crowd will feel more sympathy with this than the general public does. Basically it was that the conventional wisdom behind allowing adults to smoke, but banning cigarettes for people under 18, is wrong. Either you can believe that smoking should be permitted for everybody, or that it should be banned for everybody, but there is no consistent set of assumptions that could lead you to conclude that smoking should be banned for people under 18 but allowed for everyone else. You have two groups of people under consideration -- people under 18 who smoke, and people over 18 who smoke. What possible reason could there be for wanting to protect the health of the people in the first group, but not the people in the second group?

The problem with the conventional reason for smoking age restrictions -- "Younger people have worse judgment, so they are more likely to smoke" -- is that if this is true, all that means is that the first group of people will be proportionally larger, relative to the total population of people in their age range. But even after that assumption, you're still left with two groups of people, who exhibit the same continued bad judgment with regard to smoking cigarettes. Treating the two groups differently, is a bit like saying we should have lighter sentences for female murderers than for male murderers, just because men are more likely to commit murder.

And yet this conclusion did give me pause, so this is a classic example of an argument where you'd want someone to check your work. Off I went to create a Human Intelligence Task (HIT) on Mechanical Turk simply asking people to read the argument and respond. In the first round, most responders missed what I thought was the point of the argument, and responded with some variation of "Minors are more likely to smoke because they have worse judgment", without addressing the question of why the two groups of smokers should be treated differently. A few people responded with variations of "We've always done it that way" (referring to similar restrictions on alcohol, pornography, etc.); fair enough, it just reminded me that if I asked the question again I'd have to say I didn't consider any argument valid that boiled down to "We've always done it that way".

But then came some more interesting responses. One worker replied that I was wrong to assume that the effects of a cigarette were "the same" on adults and minors because cigarette smoke has been shown to be more damaging to developing tissues. OK, that was worth a dollar. On the other hand, that just means that there is some number N cigarettes that would be just as harmful to an adult, as 1 cigarette would be to a minor, so you're still left without a consistent reason for why you'd let the adult buy those N cigarettes but prevent the minor from buying 1 cigarette. Then another user called me out on the opening line of my original argument, "There is no reason to ban cigarettes for minors but not for adults." He said, quite correctly, that I had only attempted to debunk the most commonly given reason, but it was wrong to conclude that there was no such reason.

So, this led me to another idea for how to present an argument and solicit feedback on Mechanical Turk: in the form of a series of mathematically precise statements, each one following from the previous ones. The new HIT was to ask users if they disagreed with the conclusion, and if they disagreed, then to identify the first statement that they disagreed with. The idea was that each statement would follow logically from the ones before it, so identifying any statement as the "first" one that they disagreed with, would be tantamount to a self-contradictory paradox.

Now, whether or not you want to use this format when running an argument past the Turk workers, depends on what your goal is. If you want to really find out if your own argument is valid, then breaking it down mathematically is one approach. On the other hand, if you already believe your own argument, and you're just trying to find the most persuasive way of phrasing it, then you may not learn anything useful by breaking it down into a series of mathematical steps, because that's probably not going to be the format of our final persuasive essay.

Anyway, the new mathematical format of the argument was (slightly reworked from what I posted on Amazon):

1. Government should ban smoking by people under 18, because of the harmful health effects.
2. If that's true for the entire group of underage smokers, then it's also true for each individual smoker under 18. In other words, even if only one person under 18 smoked in the entire country, it would still be justified for the government to ban them from smoking.
3. Whatever bad health effects are caused by the average person under 18 smoking 1 cigarette, there is some number N cigarettes that would cause the same bad health effects in the average adult who smoked them.
4. If banning 1 person under 18 from smoking 1 cigarette is justified (even if they were the last smoker on Earth), and the health effects would be the same for an average adult who smoked N cigarettes, then banning 1 adult from smoking those N cigarettes would also be justified (again, even if they were the last smoker on Earth).
5. If banning 1 person over 18 from smoking would be justified, then the same logic would apply to every person over 18, which would imply banning smoking for all people over 18.
6. Hence, if you believe that smoking should be banned for people under 18, then the same logic would lead to a ban on smoking for people over 18 as well.

The response from a lot of workers who responded to this HIT was that... I lost them. Each of them identified the first statement in the list that they disagreed with, as required by the HIT, but many commented that the whole thing was phrased confusingly. There was no clear winner for the first statement that people disagreed with, but several people picked #3 and #4, arguing some version of "People under 18 have less developed judgment." (I still say that doesn't matter, because you're talking about comparing a person under 18 who smokes, with a person over 18 who smokes, and their judgment in both cases is the same, etc.) So this particular experiment failed -- it didn't make it easier to persuade people by formulating the argument as a series of steps, and it also didn't lead to any agreement on what was the Achilles' Heel of the argument itself.

However I think the general idea, of using Mechanical Turk to find sparring partners, may be useful to a lot of people. If you were interested in publishing some kind of persuasive argument, you could use an Amazon HIT to have readers compare several different versions of the same argument and identify the one that they thought was most convincing. If you were feeling more philosophical and simply wanted to know if your argument was correct, you could pay people to look for flaws in it (and here is where the mathematical phrasing could come in handy). If you're crafting an argument for public consumption, you could even have HIT workers build up your argument for you -- start with a position and have them come up with reasons supporting that position -- although to me that feels like a cheapening of the debate process that crosses the line, because you're not even trying to reason your way to a conclusion, instead starting with the conclusion you want and then working backwards (not that this isn't what a lot of debaters do anyway!). My own interest would be to see next if certain types of arguments are more likely to persuade people who are more mathematically inclined (by asking respondents to indicate how well they did at math in school). Perhaps arguments with flowery language are more likely to appeal to people who were English majors, while arguments spelled out as a series of logical steps are more likely to appeal to people who look at things in a mathematical way (also known as the "real" or "right" way of looking at things).

Maybe my preference for the controlled, user-reimbursed process of "debating" that is enabled by Mechanical Turk, has to do with a lifelong focus on bottom-line results: Decide what the result is, and judge the process by how well it brings about that result. I don't think debate and discussion should be like soccer, valued for the fun and the exercise; I think a good debate should actually get somewhere, persuading the participants or the listeners of a new point of view that builds on their old one, or else the debate has failed. If paying HIT workers kills the "spirit" of a good debate but helps achieve the goal, then so much the better. On the other hand, we'll never run out of people who enjoy the process of debating and arguing for its own sake, and will continue to debate things into the ground without anybody paying them. Hey look, here come some of them now!...

obligatorily

Do you want to have the full argument, or were you thinking of taking a course?

Re:obligatorily

Anyway, the new mathematical format of the argument was (slightly reworked from what I posted on Amazon):

Ahh...the Internet. Bringing you in touch with people who think that because they throw an 'N' into their blather, it becomes a 'mathematical' argument. Or that you can quantify things like the "health effect of smoking 1 cigarette on someone less than 18" and plop it into a faux equation. A note to the reviewer: if you're going to pay someone $1.45 to point out the logical flaws in your flawed argument (I would, but someone has already posted a rebuttal), then expect what you pay for.

Re:obligatorily

Ahh...the Internet. Bringing you in touch with people who think that because they throw an 'N' into their blather, it becomes a 'mathematical' argument. Or that you can quantify things like the "health effect of smoking 1 cigarette on someone less than 18" and plop it into a faux equation.

I heartily agree with this; there's a major tendency on Slashdot and the like to apply mathematics and mathematical logic to social arguments, but an inability in many cases for those involved to know (or care) whether this is appropriate or not.

When it comes down to it, it's an intellectual wank-fest that has little to do with the real world.

With respect to the cigarette debate, I believe that everyone missed one rather obvious point; we have to draw the line between childhood and adulthood somewhere. We protect children from things that may damage them because we don't believe that they have the maturity to trust their own judgement- yet at some stage we have to let go.

Now, you can disagree with or pick holes in what I just said when applied to the discussion in question. However, flawed or not, it's still a fairly obvious line of reasoning, and the fact that it didn't occur to any of those involved shows how far up their own pseudo-mathematical arses they were.

How I Learned Philosophy

Actually, it's not the attempt to mathify that I find problematic—I find that encouraging. It is, though, the results.

My (awesome) university philosophy professor had us do a very interesting exercise that was, though more logical than mathematical in nature, similar to what the author of TFA was going for. It goes like this...

Write down a belief that you have. For people new to this process (the entire class), this should be a strongly held belief...doesn't matter how controversial. Let's say, for example: I think abortion should be a woman's choice. (For you controversy-hounds out there, please don't mistake this for my actual belief—I'm intentionally not going to define my actual belief on this topic here.) Don't worry about getting the wording just right—you're free to revisit your initial statement as many times as you like throughout and revise it to more concisely represent your intent.

Now write down the set of "sub-beliefs" that you have which form the basis of your belief. For our example: 1. Life begins at conception. 2. Every life is equally valuable. 3. A life has no quantifiable value, but is inherently precious and ought to be protected if at all possible. Etc. Next we iterate, applying the same process to each belief listed. Obviously, you will very quickly diverge into an explosion of statements that resist corralling at every effort. Do not fret—I haven't told you about the thrust of the exercise yet.

(I should mention here that we did an entire section on identifying context-free statements, and we were asked to make our best effort to ensure that each statement was context-free, or as free of context as possible. "Context-free" means that the statement is true of our beliefs regardless of the circumstances in which the statement is tested. If that's not possible—and it's not often possible—we'd go for "generally" true, where "common sense"—whatever that is—dictates obvious exceptions.)

You will find it unnecessary to list each and every belief supporting your initial statement, which would quite likely fill several thick volumes if you did so exhaustively. Luckily, you don't have to do this to satisfy the point of the exercise, which is: where necessary, skip down to "lowest level" beliefs...that is, at some point you will mentally reach a point where you have identified a belief for which you have no further basis beliefs. When you reach this point, you have identified an axiomatic belief—that is, something you accept essentially on faith, on gut feeling, because you think it is correct. If possible, identify the key beliefs that go from your initial statement to the set of axiomatic beliefs identified.

The next step is to look at your beliefs, both axiomatic and intermediate, for consistency. In every case in carrying out this exercise, one will invariably find a whole host of contradictory statements. Then we did an iteration that attempts to resolve these conflicts by tweaking our initial statement, etc...provided we were tuning up the language to indicate real intent and not moving the statements further away from our actual beliefs, great. The ultimate idea is to identify our beliefs in all their gory, inconsistent, warty detail.

Then, we make up a list of so-called axiomatic beliefs and they are given to 5 random classmates (all double-blind, of course). You then are tasked with taking home those 5 lists of axiomatic beliefs and attempt to drill down further. If they are truly axiomatic, you won't be able to do this—the idea here is that you ultimately get back 5 people's analysis of your list and given another chance to continue the process—most of the time, it turns out you realize your axiomatic beliefs weren't axiomatic for you after all, and that you can actually drill down even more.

Anyway, it goes on like this, the ultimate point being that you arrive at some network of beliefs which you apparently do accept as axiomatic. The focus here is not on the logic that leads you down the path from the initial statement to the final list...the point is to show that your beliefs are not rigorously logical, even after you've done your level best to identify all the logical flaws, ultimately you wind up with a list of axiomatic beliefs that either directly or indirectly contradict each other to some extent. What these beliefs are, where the conflicts are, and how you resolve these conflicts all roughly correspond to your worldview.

As an entertaining add-on at the end of the course, the prof provided us with some very mild statistical metrics that told us how self-contradictory our beliefs were when pegged against our classmates, previous years, different types of statements (it was generally true that the more strongly held / the more controversial the statement, such as the abortion example above, the less self-consistent the foundational beliefs identified).

For a couple of years after doing this exercise, I found it very difficult to make strong statements of opinion about controversial topics. My mind would involuntarily start this process, identifying all the biggest logical hurdles and inconsistencies built into the statements I was about to make. This reflex also made me annoying to others with strongly held beliefs. :-)

nope.

"Hell, you can find enough people on UseNet and Slashdot who will argue with you just for free."

no you can't.

Re:nope.

Now you're just contradicting.

Re:nope.

No he's not.

Here's a HIT task

What the fuck are you blabbering about? Is there any way you can summarize it into 100 words or less?

Er, what?

No offense, but your argument's not that good, or maybe I'm not understanding your point.

-You just "proved" that no paternalistic intervention is ever justified, *even by parents to newborns*. Hey, if you believe compelling someone to eat is okay if they're under 2 years old, obviously, there must be some insufficient amount of eating you can do when over 2 years old that would justify force-feeding. Er, yes, there is, it's just not encoded in any specific law that way.

-The "judgment" argument is completely unrelated to the "health results" argument (up to a limit). You seem to think the argument is that

"People under 18 shouldn't be allowed to smoke, because if they did, they would smoke a lot due to bad judgment, and people over 18 would not excessively smoke due to bad judgment."

  It's not. It's more like,

"People under 18 shouldn't be allowed to smoke because their poor judgment makes them unable to accurately weigh long-term consequences of smoking. Therefore, they will smoke, and later regret the poor health and addiction. Adults may do it in the exact same amount, but then it would be with accurate judgment of the consequences. The rational self would not be victimized by the previous irrational self."

-You perform a reductio saying that banning smoking for minors would imply banning some amount of smoking (N) for adults. There is such a ban, so there's no contradiction. Namely, if you smoke so much at once as to nearly kill yourself, that can be considered a suicide attempt, and people can legally restrain you from doing it further until your body can cope.

(I'm not saying 18 is right age to ban smoking. I'm not saying there should be any one age. I'm just saying that this is a poor representation of the case for banning underage smoking, and a poor argument for a change in policy.)

Now, give me my $1.

Re:Er, what?

Also, the assumption that if 1 cigarette will cause health effect X to a child under 18, then N cigarettes will cause the same health effect to an adult is false correlation. There are many processes occurring in a growing child, from the initial growth of nervous tissue, to the rapid replication of cells from the stem cell forms, that are significantly reduced or even eliminated when the person reaches adulthood. Damage can be caused by a growing person smoking that is impossible to replicate in an adult.Likewise, cigarettes have a different effect on the systems of a grown adult that are not necessarily present in a child.

If you are going to use an equivalence class in your argument, you must make sure that for all cases related to the argument that the equivalence holds true.

Re:Er, what?

What's surprising is that nobody pointed out the most obvious reason why this person's arguments were wrong. Even if you ignore peer pressure and susceptibility to addiction, it is an undeniable fact that nicotine is an addictive stimulant.

The key argument in the line of logic was that the adults who start smoking later are exercising the same bad judgment as those who continue to smoke having started earlier. This conclusion, however, is fundamentally flawed. The people who start smoking earlier and are merely continuing to smoke suffer significant physical and psychological stress when they attempt to quit, causing them to start again.. People who never smoked before and start smoking do not start smoking because of such physiological symptoms. Thus, starting smoking later in life, while requiring the same poor judgment as someone starting earlier in life, requires substantially worse judgment than someone continuing to smoke.

Thus, if we assume that the judgment norm improves with age, someone starting to smoke earlier in life can be typical for his/her age group, while someone starting to smoke later is exhibiting atypically poor judgment. You can't protect everyone, and it makes sense to focus protection efforts on people within a couple of standard deviations from the norm. Therefore, setting a cutoff age of 18 or 21 is effectively saying that by the time you reach that age, the judgment of people within that range of the norm should be sufficiently developed that they will not start and thus will not become addicted.

Another fallacious key argument is that what applies to a group as a whole must necessarily apply to each member of the group. Let's say you have 1% of the population who is known to be immune to a contagious disease. Assume that this disease starts spreading rapidly through the population. Do you A. quarantine everyone for their own protection, or B. quarantine only the 99% people who could catch the disease? For the 1%, it is not truly for their protection, but you quarantine them anyway to avoid them carrying the disease. However, if the reverse were true---if 1% were at risk and 99% were not---the more reasonable solution would be to quarantine the 1% who were at risk. For example, you might recommend that small children and the elderly stay in their homes and limit contact with other people until the outbreak has been stemmed. There would be no practical reason to inconvenience everyone else for what amounts to a minimal decrease in the risk for that 1%.

Similarly, the assumption that it should be illegal even if only one person smoked is a really huge fallacy. Selling cigarettes to minors should be illegal only if the benefits of having the law on the books outweigh the enormous cost of having to maintain the additional legal framework for dealing with the controlled substance. If you passed a federal law to prevent exactly one person from smoking with full knowledge that no one other than that one person would ever smoke, you would be branded a complete moron. These laws are on the books because of the presumption that there is a significant gain by preventing this behavior across a larger population, not because it would be useful for preventing one person from smoking. The odds are that some portion will fall through the cracks and smoke anyway, and to a large extent, it isn't worth the effort to combat these edge cases unless they become statistically significant for a given location (e.g. a store that routinely sells to minors). Passing stricter and stricter laws rapidly becomes a question of diminishing returns.

My personal opinion is that smoking in public places where other people have to breathe it should be banned. There's the big public health threat, not smoking itself. People should be allowed to kill themselves if that is their choice, but the moment they inflict it on innocent bystanders, I have a problem with it. I also support insurance companies charging more money to insure smokers so that we don't all have to p

Nah.

I don't need to pay someone to argue with me. I have children.

An Argument

Here's the first argument I thought in favor of the under-18 ban. It has nothing to do with "people under 18 have worse judgement".

(1) Adults inherently have more rights, and are expected to shoulder more responsibility, than those who are minors (e.g., voting, driving, running for office, registering for military service, medical care, curfews, etc.)

(2) Granted that smoking is bad, we have considered banning it for everybody.

(3) It's been decided that this is not so critical an issue that it trumps adult rights and self-responsibility; therefore we feel it would be improper to make a ban for adults. However, it does rise to a level above the threshold for minor rights, and therefore a ban for minors is considered an acceptable prohibition.

Why pay for what you can get for free

Linux is a terrible, insecure operating system with a higher TCO than Windows.

Bill Gates is more enigmatic than Jobs and more intelligent than Stallman
The MPAA/RIAA are well within their rights when they sue people.
Darl was right.
I love the PS3 and hate the Wii

Discuss!

Re:Why pay for what you can get for free

great. now we have an article where troll comments are on-topic.

you did all this for a bunch of virgins?

are you kidding me? i'd kill you for a klondike bar.

Free answer to the question

The benefit of a ban is possibly improved health in the group of people the ban effects.
The cost of the ban is decreased freedom in the group the ban effects.

Your argument address the possibility that the value of the benefit might differ with age.
You argument fail to address the possibility that the value of the cost might differ with age.

Since society value freedom for adults much higher than freedom for minors, the age discrimination is justified in this case. QED.

(A totally different discussion is whether society should value freedom for adults higher than freedom for minors, but it clearly does).

Here's one for the Mechanical Turks:

Why did Constantinople get the works?

Mind made up?

It's interesting and surprising to me that you do not even consider the possibility that a rebuttal may change your mind. You seem to assume that the only thing that will happen is that you will change or improve your argument in order to strengthen it. Apparently if you have some reason for your belief, and you find that there is a very effective rebuttal, you will simply avoid citing that reason in your future arguments; but knowing that one of the reasons for your belief was wrong will not make you less likely to hold it.

It's a pretty sad commentary on the state of our reasoning process and makes one wonder why people even try to come up with arguments. I guess they can be seen as tools for closed-minded people like the OP to persuade the open-minded (which the arguers apparently consider weak-minded).

Re:Must..resist....

Shamelessly stolen from http://www.mindspring.com/~mfpatton/sketch.htm [mindspring.com]:

Q: WHAT DO YOU WANT?
M: Well, I was told outside that...
Q: Don't give me that, you snotty-faced heap of parrot droppings!
M: What?
Q: Shut your festering gob, you tit! Your type really makes me puke, you vacuous, coffee-nosed, malodorous pervert!!!
M: Look, I CAME HERE FOR AN ARGUMENT, I'm not going to just stand...!!
Q: OH, oh I'm sorry, but this is abuse.
M: Oh, I see, well, that explains it.
Q: Ah yes, you want room 12A, Just along the corridor.
M: Oh, Thank you very much. Sorry.
Q: Not at all.
M: Thank You. (Under his breath) Stupid git!!

Re:Must..resist....

Oh I cant help it! This reminds me too much if Monty Python's Argument Clinic

No it doesn't.

Re:Must..resist....

This is not even an argument!

Re:Must..resist....

"This is not even an argument!"

Yes it is!!

Re:What are you smoking

and there's apparently no mention of the fundamental premise to this restriction: people under 18 can't be trusted to make their own decisions.

Not quite; people of any age are allowed to make their own decisions.

The thing that happens when you're 18 or older, though, is that you then have to be responsible for your decisions. When you're less than 18, the consequences get handled by society at large.

That's the difference that sets the line for smoking at 18. Doesn't have anything to do with health issues except for who has to pay the majority to fix those issues. (Here's a hint: what tax burden is paid by those younger than 18?)

Re:What are you smoking

Here's a hint: what tax burden is paid by those younger than 18?

Answer: About $9085.6 billion + $1.4 billion/day. That's the amount that the generations currently in charge (including me) have spent, but not paid for. We show no signs that we plan to paying it, ourselves, any time soon. We had better hope those kids exercise better judgment than we have and pay that number down.

Are you suggesting that our banning smoking by 18-year olds is a rational economic response to the minuscule amount of taxes we let ourselves pay? Horse-cookies! If we were rational economic actors, there are a lot of bigger-ticket items we would have eliminated first.