Turing Test 2: A Sense of Humor

mhackarbie writes "Salon has a great story, Artificial Stupidity, about the Loebner Prize, a yearly contest that for over 10 years now has offered a $100,000 prize to anyone who can create a program to pass the Turing Test. The best part is the resulting fiasco that develops between the eccentric philanthropist who started the contest and extremely annoyed AI Researchers such as Marvin Minsky."

well ...

All Hugh Loebner wanted to do was become world famous, eliminate all human toil, and get laid a lot.

does this mean we're all considered entrepeneurs ?

What about people who fail the Turing Test?

I don't think bots are the problem... I've had several online conversations which I'd assumed were chat-bots but turned out to be real people. I guess when Turing designed his test, he probably didn't anticipate the massive advances in human stupidity that we've witnessed in the last few decades :)

Re:What about people who fail the Turing Test?

You post is marked "Score:5 Funny", but I would mark it "Insightful".

That testers can believe that humans are computers is why it will never be a 'test'. Turing himself only ever called it the 'Imitation Game'.

If there is no way to tell humans from computers, how can you ever tell the computers from the humans?

We likes the 'turing test' not because it is scientific, but because, like intelligence itself, it is ill defined and imperfect.

I love the Loebner quote: "My reaction to intelligence is the same as my reaction to pornography, I can't define it but I like it when I see it."

petulant prima donnas and inconsistent judging...

Every person with whom I spoke about it said that last year's contest was an utter fiasco, with unclear rules, inconsistent judging, arbitrary fiats by an opaque prize committee, petulant prima donnas, and last-minute changes of venue that prevented most entrants from even discovering where the contest was taking place until after it had happened.

Was it held in Florida as well ? Or is it just a massive coincidence ?

Re:The most interesting thing...

for years and years and years, researchers have been promising AI was just around the corner... And what do we have right now? Nothing!

Get with the program, dude: we had AI even in the UK last year. I didn't go and see it though, because it starred that irritating kid from "The Sixth Sense".

Re:The most interesting thing...

This strikes me as true: for years and years and years, researchers have been promising AI was just around the corner... And what do we have right now? Nothing!

Well, nothing is a very relative term. We now have AI capable of counting the number of cars on a given street given a photograph of a region, and can automatically follow people / vehicles / animals as they travel around and through objects. OCR is accurate enough to be implemented professionally, and voice recognition is up to 95%. None of these were possible 25 years ago, and not just because of a lack of hardware.

While full AI is still a while away, the first major stumbling block, pattern recognition, is well on its way to being solved.

The AI in Quake 3 is much better than the AI in Pong.

-C

Re:The most interesting thing...

Part of the problem is that things that were once considered part of AI have moved out and become mainstream technology. Voice recognition, Expert Systems, Fuzzy Logic, Neural Nets, Chess playing computers...all of these were once considered to be unsolved AI problems but since they are now in common use, we don't consider them a part of AI anymore.

You can find plenty of twenty to thirty year old textbooks that tell you that playing chess at grand master level would be a sign of computer intelligence - now we know that all it takes are some clever heuristics and a lot of CPU power.

As soon as computers can pass the Turing Test, it'll be considered laughable that anyone ever thought it required *intelligence* to chat with a human. In a sense, this has already happened. Quite a few people were convinced by Eliza - but you can tell from just looking at the code that it's not intelligent.

The same thing is happening with animals. We used to define humans as the only tool-using animals - then they found birds breaking open clamshells by dropping rocks on them. The definition changed to humans as the only tool *making* animals...then they found chimpanzees who strip the leaves from twigs before they poke them into anthills. So then it was 'self recognition' - that also failed with dolphins who can recognise themselves in a mirror. Now it's some other thing. Animals will never be labelled intelligent" because the definition of intelligence is that thing that humans have but animals do not.

I predict that we'll never have AI. That isn't a failure of the work - it's in the nature of our definition of Intelligence as "that thing that humans have that animals and machines don't have".

hmm, well

some people [arizona.edu] and their followers [innerx.net] do not believe that machines will EVER achieve human level intelligence.

(overall a good read. certainly a buttload of speculation but no more (actually probably less) than found in Wolfram's book)

On the other hand, I see nothing wrong with offering a prize for what he believes in. Heck we have the Templeton prize out there (more than the Nobel, no less) for best achievement in religion (christianity specifically, methinks), so what's wrong with offering 100G of his own money? We also have the X-BOX cracking contest - who is willing to bet that the believing in the chance of solving a 2048bit key in a few monthes is MUCH dumber than trying to shoot for some "not everybody agree as AI" AI?

The Meta Turing Test

Any specified Turing Test can be defeated in much the same way as a lock-pick can defeat any specified lock, so perhaps we should move up one level of abstraction. I propose the "Meta Turing Test" which is as follows: specifying the conditions of the Turing Test (ability to lie, sense of humour, etc.) should allow a true human to design an automaton that fools the turing test, while a computer will not be able to do so.
Alternatively, why not just abandon the myth that human intelligence is some kind of mystical cloud, and see it for what it is, namely a set of thinking organs each designed (or adapted, if you prefer the 'evolution is a passive process' concept) to solve specific problems, in the same way as my hand is adapted to handling objects. Then, test each of these tools carefully. Anything - computer or human - that passes the tests can be defined as 'human'. Many beings that we today consider human will probably fail. Borg borg.

Why are they upset?

Why are Minsky and Shiber so upset that a sex-addicted pothead is sponsoring an A.I. prize, when the Father of Dynamite [northpark.edu] sponsors a Peace prize?

Loebner can do whatever he wants with his dough. No one is being coerced into entering his contest.

Re:Why are they upset?

That is not fair. Minsky might have percieved that this was the case, but it doesn't follow that it was. Loebner gave a perfectly good explanation for the clause (See below) and it seems pretty hypocritical that Minsky fumes that Loebner uses his name as co-sponsor in advertising ;)

From: [e-mail] (Hugh Loebner)
Newsgroups: comp.ai
Subject: Minsky Co-sponsor of Loebner Prize!
Date: 8 Mar 1995 16:48:36 GMT
Organization: ACM Network Services
Lines: 63
Message-ID:

In Message ID Minsky writes:

>In article [e-mail] writes ....
>>17.The names "Loebner Prize" and "Loebner Prize Competition" may be used by
>>contestants in advertising only by advance written permissionof the Cambridge
>>Center, and their use may be subjecttoapplicableicensingfees. Advertising is
>>subjecttoapprovalbyrepresentativesoftheLoebn e r Prize Competition.Improper or
>>misleading advertising may result in revocationoftheprizeand/or other actions.

>[Some words concatenated to enforce the 80-character line length
>convention.]

>I do hope that someone will volunteer to violate this proscription so
>that Mr. Loebner will indeed revoke his stupid prize, save himself
>some money, and spare us the horror of this obnoxious and unproductive
>annual publicity campaign.

>In fact, I hereby offer the $100.00 Minsky prize to the first person
>who gets Loebner to do this. I will explain the details of the rules
>for the new prize as soon as it is awarded, except that, in the
>meantime, anyone is free to use the name "Minsky Loebner Prize
>Revocation Prize" in any advertising they like, without any licensing
>fee.

1. Marvin Minsky will pay $100.00 to anyone who gets me to
"revoke" the "stupid" Loebner Prize.

2. "Revoke" the prize means "discontinue" the prize.

3. After the Grand Prize is won, the contest will be
discontinued.

4. The Grand Prize winner will "get" me to discontinue the
Prize.

5. The Grand Prize winner will satisfy The Minsky Prize criterion.

6. Minsky will be morally obligated to pay the Grand Prize
Winner $100.00 for getting me to discontinue the contest.

7. Minsky is an honorable man.

8. Minsky will pay the Grand Prize Winner $100.00

9. Def: "Co-sponsor": Anyone who contributes or promises to
contribute a monetary prize to the Grand Prize winner .

10. Marvin Minskey is a co-sponsor of the 1995 Loebner Prize
Contest.
-------------
BTW

The language that Minsky finds so offensive was added
by the Prize Committee because of a possible mis-representation
regarding the contest made by an annual prize winner.

No fees have been requested of any winner, nor do I anticipate
of any fees ever being requested. Rule 17 merely protects the
Loebner Prize from misrepresentation in advertising.

Re:Why are they upset?

> Rule 17 was not the only example of Loebner and co. being pompous and humorless.

I've looked over the article and some of the transcripts. It seems pretty obvious to me who gets the mantle of 'pompous and humorless.'

Minsky's best attempt at humor was his $100 'prize', and Loebner turned that around and made it bite him so hard that I doubt the man will ever attempt humor again. Which is okay, I guess... it was amazingly pathetic and meanspirited even before Loebner hit him over the head with it.

Basically, you have a person who everyone in the field thinks is a god. Is it any wonder that everyone in the field thinks that every time he opens his mouth, whatever he's arguing against is successfully demolished? They don't even have to listen to whatever he's saying... I mean, how often does God get out-argued in the bible? Can't happen. Ignore all evidence to the contrary. I guess it's not even surprising that his arguments don't hold water... if you've been considered a god for a while, your 'intelligent argument' muscles start to atrophy. And no matter what anyone says, those are diffreent muscles than the ones you flex when you're thinking about how to set up a new kind of neural net.

It seems to me that Loebner has his points. You may not agree with them, but at least try to find sound reasons for disagreeing. Saying that HE is humorless and pompous, when Minsky has laid nearly exclusive claim to that particular high ground in the conversation, just makes you look, uh, humorless and pompous. And maybe a wee bit... dumb?

-fred

Bloody-Mindedness

"... extremely annoyed AI Researchers ..."

Perhaps "extremely annoyed" is what distinguishes human intelligence from machine intelligence?

In John Brunner's non-novel Stand on Zanzibar [sfreviews.com], cranky sociologist Chad Mulligan declares that supercomputer Shalmaneser is now intelligent because Shalmaneser has displayed the quality of "bloody-mindedness". Not the same as "annoyance", of course, but in the same emotional realm ....

Missing the point

I think that people who focus on the Turing test are missing the point, this isn't really AI and probably doesn't have much of a use outside advertising to via IRC/personal messaging etc.

The real interesting areas of research in AI are for example: in dye-master processes, where AI replaces a highly skilled human, or automating the driving of cars. These are all AI and, IMHO, much more impressive than glorified Eliza, Turing test stuff...

Re:Missing the point

I have the distinct feeling that "worthy" AI objectives are defined by the AI community as "those things we think we can do reasonably well at the moment."

Not hardly. As it turns out, one of the more frustrating aspects of AI is that once some particular computation that would appear to be correlated with intelligence can be performed, then it invariably doesn't count as AI anymore. So there are lots of practical systems out there today that can prove theorems, do symbolic algebra, play chess better than 99.999% of all people, a whole bunch of stuff. But hardly any of this strikes us as AI anymore. On the other hand, there are lots of horribly difficult problems out there whose solutions we really can't expect to get at within 10 years, and those are all "good" AI problems. Now, one thing that makes them good problems is that we know they contain many different thesis-sized projects that correspond to sub-goals for the "real" problem, and because it is possible that knocking off some of these subgoals could yield some real insights.

Now the interesting thing to notice here is that Turing was a *very* smart guy, and any program that successfully passes the strong version of the Turing Test has almost by definition solved every hard problem that confronts AI, and all of the subproblems that compose those problems, and... It's a truly gargantuan task, and one where even your most advanced programs are almost guaranteed to look really bad in competition.

Having said that, I do still think there is some point in holding contests like the Loebner, not for what they will tell us about the state of how fast AI is progressing, but because the programmers who compete at this point really are trying to scam the system and "get away with" producing a program that is NOT intelligent but that might LOOK like they are intelligent. Understanding how clever these deceptions can be, and why we fall for them, is itself an interesting by product of the competition. So the importance of ELIZA in the end was not that it was a great piece of code or introduced techniques that we could build on directly, but because it taught us a *lot* about people's implicit assumptions about a conversational partner, and how you could generate conversational situations that could finesse the hard stuff. So people don't go out to talk to ELIZA with the goal of determining that it is just a program; they don't go looking for the disconfirming evidence. That's a pretty key point in itself.

The designer needs a sense of humour too

Let's face it, if the designer of the program doesn't have a decent sense of humour then his program is likely to fail. I take this opportunity to remove a fair chunk of the contestants by posting "The world's funniest joke" If you don't laugh then don't enter:

Two hunters are out in the woods when one of them collapses. He doesn't seem to be breathing and his eyes are glazed. The other guy takes out his phone and calls the emergency services.

He gasps: "My friend is dead! What can I do?" The operator says: "Calm down, I can help. First, let's make sure he's dead." There is a silence, then a gunshot is heard. Back on the phone, the guy says: "OK, now what?"

A sense of humour?

Computers passed that test years ago. I mean, who can forget the classic:
keyboard not found, press F1 to continue

Cheers,
Ian

The program that passes the test is :

the program that alters the test to fit its own capabilities. That is cheating? How more human can it get ? Humanity is constantly adapting it's surroundings to fit its own needs...

Consciousness

Physics of Consciousness

Building a machine to pass the Turing Test is one thing, but the nature of consciousness itself is the more profound question here. Rodney Brooks asked this question in a relatively recent Edge Online interview [edge.org].
What are we missing in our computational models of living systems?

Chris

http://www.umsl.edu/~altmanc/ [umsl.edu]
http://www.artilect.org/ [artilect.org]

The "Turing test" was a joke

The author of the article appears never to have read the article by Turing where he described the so-called 'test'. It is clear that Turing was a deep and subtle thinker way ahead of his time. If you read what he is saying in context, he is arguing that first and foremost, thought can be automated in the sense of a universal computer which can compute anything that a brain can. To his critics who said that this was somehow impossible, he created a reducto-ad-absurdum argument; he said look if you are talking to this machine and it is composing sonnets which are like Shakespeare, and you *still* can't say it's intelligent, then you are an idiot. He was not proposing that this was an objective test or a desirable thing to do, he was poking fun at idiots like the author of the Salon article.

Re:The "Turing test" was a joke

*ding ding*

Turing wasn't looking for a UNIVERSALY INTELLIGENT MACHINE, he was looking at how machines could act intelligently. We're not talking about human in a computer, we're talking about can a computer act intelligently. If you think it's impossible, tell that to people that can be "fooled" by bots on IRC or MUDS for weeks or more.

Seriously, we're obsessed with the idea of human intelligence, which is often times an oxymoron, but that's what we want...

Sundman

John Sundman who has written this article has also written a quite interestion book called Cheap Complex Devices (he mentions is in the article).

It's kind of wierd and strange - the idea is that the novel was one of two novels written by a computer program.

I've reviewed it here [sfbook.com].

Academic AI is a con game

I was active on comp.ai at the time Minsky made his offer [Google query] [google.com], and I'm convinced the real reason academic AI hates the Loebner Prize is that it shows up how little they've managed to accomplish.

I agree that the entries are really bad-- one recent winner just said the same things no matter what the human asked. But one winner, unmentioned in Salon, was Thom Whalen [dgrc.crc.ca], whose design was a genuine advance in the art. (Regrettably, Loebner changed the rules to exclude his approach in the future.)

What Whalen did was limit his domain to one topic, and compile a set of general answers to likely questions, which he matched by spotting keywords. So even if the answer wasn't a perfect match, it was general enough to be useful. This design should be better known and more widely used, and the Loebner contest would have been a good launchpad to bring it to people's attention if the academics weren't so prejudiced.

But the top academics get six-figure salaries for generating lots of jargon and no useful products, so a level playing-field is the last thing they want.

Re:Academic AI is a con game

How is what you describe an advance in AI?

It's a paradigm shift-- instead of looking for complicated 'solutions' that will enhance their status, Whalen took a fresh look at the problem and found a way to deliver useful results with no particularly fancy algorithms.

It's nothing that anyone couldn't already add to a system that needs it.

No one had at the time, and few are even aware of the idea now.

Check out TRECK

The dismal website design shows how little they appreciate Whalen's insight-- I clicked four different links on the homepage and ended up with ZERO examples of their work. This is absolutely typical of academic-AI websites-- a whole lot of self-congratulation and almost no effort to communicate. (Contrast that with any healthy science, where tutorials aimed at beginners are a dime a dozen.)

To say AI has made no advances because we can't fool people into thinking they're talking to someone

Those words are yours. Academic AI has made a few minor advances, but continues to project itself as possessing arcane, complex secrets that deserve big paychecks.

Why the contest rubs AI people the wrong way

Turing defined the test more than 50 years ago. Considering that there were barely any machines at that time that we would call computers today, his prescience was remarkable.

Turing stipulated in the Turing test (TT) that the "interrogator" specifically has the goal of trying to determine which of the contestants is human and which is the machine. Unfortunately, the way the Loebner contest is conducted, this important requirement is completely ignored (at least in the default $2000 prize). As a result, the results of the contest are completely irrelevant from the point of view of the Turing test. Claiming otherwise is incorrect and misleading, and Loebner fully deserves all the criticism he gets.

The TT is still fully valid today. We are very far from building bots that will pass it. (though Turing predicted that by 2000 we will have machines that will pass TT). In fact, the whole direction of work on the bots participating in the current day Loebner contests is irrelevant from the TT point of view. They work mostly by building enormous databases of statement-response pairs and doing minimal reasoning. Turing would have died laughing if he had known people would take this approach to passing the TT. Let me illustrate why the database idea is insufficient by itself: for a bot to pass the real TT, it would have to answer questions like "what is the integral of e^x dx". Remember that the interrogator is actively trying to find out if it is a human or a bot. The objection "but two humans in conversation wouldn't ask such question" is invalid, and this is precisely why the Loebner contest is stupid.

The reason why today's bots are so unsuccesful is not far to seek. It has long been known in the AI community that get anywhere near passing the TT, a bot would need what is known as "world knowledge". To build world knowledge, you need memory approximately the capacity of the human brain: estimated to be the order of a petabyte. And processing power to match: the brain runs something like a billion threads in parallel, and is 10^7 times as energy efficient per computation as today's computers. Of course, we aren't there yet. Thus, contrary to what most people would feel the thing that is holding AI up is hardware.

Similar to today's bot craze, there have been crazes in the past when people thought they were close to building truly intelligent machines ("expert systems" comes to mind.) However, they inevitably came up short because the hardware power wasn't there. In about 20-30 years, assuming there continue to be breakthroughs in storage technology to keep up the doubling, computers will be matching the brain's capacity, and then we'll be talking.

Summary: to hell with people who apparently popularize science and end up giving the real researchers a bad name.

The Best Part

In 1995, about a year after the publication of Shieber's article, Marvin Minsky, the father of artificial intelligence, posted a notice on the comp.ai and comp.ai.philosophy Usenet newsgroups. In it he drew attention to a clause in the Loebner contest rules to the effect that using the term "Loebner Competition" without permission could result in a revocation of the prize.

Minsky wrote, "I do hope that someone will volunteer to violate this proscription so that Mr. Loebner will indeed revoke his stupid prize, save himself some money, and spare us the horror of this obnoxious and unproductive annual publicity campaign. In fact, I hereby offer the $100.00 Minsky prize to the first person who gets Loebner to do this. I will explain the details of the rules for the new prize as soon as it is awarded, except that, in the meantime, anyone is free to use the name "Minsky Loebner Prize Revocation Prize" in any advertising they like, without any licensing fee."

(Minsky did not respond to e-mails requesting an interview.)

If the CACM article marked Loebner's fall from grace, the Minsky note on comp.ai marked his utter banishment into the wilds of A.I. quackery.

Can you imagine, for example, being a graduate student in computer science at a big-name school in 1996 and telling your major professor that your goal was to win the Loebner? Loebner was more "out" than Liberace.

But Loebner did not take his snubbing meekly. Loebner immediately wrote back that the best way for Minsky to get Loebner to revoke his prize was to win it. Of course Minsky had already hinted that Loebner had never made clear what the rules for winning the prize were, so that was not a very satisfactory rejoinder. But then a few days later ("while taking a nice hot bath, drinking a fine wine, about an hour after smoking a really fat joint"), Loebner came up with a more considered and clever response, one that still rattles Minsky nearly a decade later.

Minsky had announced that he would give $100 to whoever made Loebner stop his contest. But Loebner would only stop his contest when somebody won the gold medal. Therefore, Loebner reasoned, Minsky, being an honorable man, would give $100 to whoever won the ultimate Loebner competition. Therefore, Marvin Minsky was a cosponsor of the Loebner competition, simple as that. It was delicious!

Loebner promptly issued a press release saying that Marvin Minsky was now a cosponsor of the Loebner Prize, by virtue of his announcement of the "Minsky Loebner Prize Revocation Prize." What made this development so delightfully ironic was Minsky's own statement that anyone was free to use the name "Minsky Loebner Prize Revocation Prize" in any advertising they liked, which made it nearly impossible for Minsky to prevent Loebner from doing just that. Which is why Loebner continues to cite Minsky as a cosponsor of his event every chance he gets.

The image that comes to my mind whenever I think of this development is from the sublime cartoons of the late, great Chuck Jones, with Hugh Loebner in the role of Bugs Bunny, and Marvin Minsky, the father of artificial intelligence, in the role of Yosemite Sam, stamping his feet, with smoke coming from his ears. In fact, Minsky is still listed as a cosponsor of Loebner's prize on the Web site, and, as we'll see, Minsky is still stamping his feet.

AI is a fraud

I worked in a research lab that shared a building with MIT's artificial intelligence laboratory. And I have to agree with the article. The AI field is a fraud. Again and again, there would be big placards in the lobby announcing gala media events up in the AI Lab. (We lesser mortals dutifully clomped upstairs to eat the expensive, catered food.)

And yet *nothing* *ever* *happens* in the field.

Every now and then a new "hero" emerges. For a while it was Minsky. In recent years, it has been Rodney Brooks. Regardless, you can see the current hero on TV all the time, commenting on matters as an "AI expert". They don't tell you that Brooks' course is widely viewed as a complete crock; a few puerile algorithms, some linear differential equations, some finite automata, and THAT'S IT. The rest is all blabbering with no substance.

The AI community uses rotating hero-worship in lieu of progress. But it isn't like any of these guys is an actual "AI expert". There are no "AI experts", because there is no such thing as artificial intelligence in this world. They are no more experts on AI than I am an expert on Martian fruit exports. In this field, you don't need real research; an Australian accent and good sense of humor suffice.

True artificial intelligence would be amazing. But the field has made essentially zero progress in the last fifty years. Obviously, it is a really hard problem. On one hand, the AI guys do what other fields do when they're stuck (since they *must* continue to pump out graduate students, attract grants, etc.), they keep trying to change the question. But the pathetic thing is that many completely denigrate the most obviously fair benchmark-- the Turing test.

Coincidentally, a benchmark showing the complete failure of the field.

Re:AI is a fraud

Dude, sharing a building with AI types an expert on AI does not make.

Examples of advances in AI:

1. Computer programs able to spank all but the best humans playing chess.

2. Computer programs able to spank your ass playing even more complex games like CIV 3, C&C, etc.

3. Google saying "Searching 3,083,324,652 web pages" and "Results 1 - 10 of about 1,500,000. Search took 0.07 seconds"

There's been huge advances in AI with such things as Genetic Algorithms and Fuzzy logic. The applications are very specific and are not the far reaching HAL 9000 that people traditionally think of when you say AI. There is no 'singular consciosusness' that is going to pop out of your computer. That is NOT what AI is about. AI is about solving problems. More specifically, it's about finding methods for a computer to solve problems without brute forcing them.

For example, it would be easy for a computer to beat a chessmaster if the computer had the whole search tree available. The out come of every move of every game would be available, and it would be trivial to steer it towards a victory. But since the tree is HUGE and would take many hundreds of years to generate, the problem of computers playing chess is to get them to figure out a 'smart' way to beat the chessmaster. Alpha-beta tree pruning and things like that are the results. Don't underestimate the power of these.

There are great things coming out of AI research all the time, but you will not be seeing HAL 9000 any time soon.

If there is any existing program...

that has a chance in hell, I'm placing my bets on