Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Programming IT Technology

Turing Test Competition At CalTech 173

Posted by timothy from the move-near-a-major-target-first dept.
Charles Dodgeson writes "The Turing Tournament at Cal Tech wants to know if you can program an emulator that will play games like a human, or if can you write detector that can correctly sort the wetware from the software. Before you get too excited, the "games" are very limited things. But there is a $10,000 prize for the winner. You can read the gory details."
This discussion has been archived. No new comments can be posted.

Turing Test Competition At CalTech More

Turing Test Competition At CalTech

Comments Filter:

  • Programs to play games and programs to catch them (Score:3, Funny)

    by JHandey ( 635824 ) on Tuesday January 14, 2003 @10:07PM (#5085048)
    Exactly how not needed do we want to make actual people?
  • "[The competition] is about as productive as promoting biology by offering a prize to the designer of the most convincing silk flower..." - Steven Pinker

    What does this contribute to the field?

    • Re:useless (Score:3, Interesting)

      by Charles Dodgeson ( 248492 )
      This is not the Loeber prize, which Pinker was describing correctly. This is a much more constrained and narrow contest, focusing on a very well studied portion of human behavior.

      Does anybody actually read the stuff pointed to before posting?

      This actually is useful, but not for AI. There is a whole branch of what is called "experimental game theory". Getting something that plays these games like humans is interesting (well, to me at least).

    • Re:perhaps not... (Score:2, Insightful)

      by dtgm01 ( 636096 )
      haha, I like Pinker. Having the support of a tower of feathers, ai work has fallen to the trenches With as much overhype as people have believed people are looking at it differently [melbpc.org.au], this largely being the case of people who work in the field. Perhaps it is just not it's time yet [guardian.co.uk] to be introduced to the market Now, from the article [caltech.edu]...

      The Turing tournament is a two sided tournament designed to find, on the one hand, the best computer programs to mimic human behavior, and on the other hand, the best computer programs to detect the difference between machine and human behavior. Two types of submissions will be accepted: an emulator [caltech.edu], which mimics human behavior, or a detector [caltech.edu], which detects the difference between human and machine behavior.

      So, I suppose we could say by evaluating the success of response (as would be weeded out by whomever *actually* turns out an entry), we will have achieved our research, VOILA! It's a successful research incentive, the prize that is.

      Whaddya think? no? heck of a fight though wasn't it? :P

      • In regards to people are looking at it differently; I meant the general publics views on ai disrupting the entire economy and life in one swift sweep and all. While this may be true, it most likely won't come tomorrow, and it seems as though more and more people are starting to understand this. At least this is what I see and what I pathetically tried to convey in the previous post. (you can throw the tomatoes now)

        The guy of the post before me (I say guy because of the tone I got from the post, what? I can't use common sense anymore to form a deductive statement based on the US' political correctness? bs) had a good point of the other applications such a contest can provide value to.

  • Translation, please. (Score:5, Funny)

    by Anonymous Coward on Tuesday January 14, 2003 @10:10PM (#5085064)
    Would someone please translate this [caltech.edu] to something than can be parsed by non-math types?
    • I tried but I got stuck on Step 1!

    • Re:Translation, please. (Score:5, Informative)

      by Jason1729 ( 561790 ) on Tuesday January 14, 2003 @11:08PM (#5085345)
      Step 1:
      Make up a set of game boards and have a group of humans each play the game on those boards. Each human will play once on each board. This gives us real human data to compare the software to.

      Step 2a:
      Let each of the submitted emulators play the game on every one of the boards created in Step 1. We now have a set of results for each human and each emulator on all the game boards

      Step 2b:
      For every detector that was submitted, give if every set of results. It returns its answer for which it thinks are humans and which are emulators in a very precise way. We now have a matrix of (number of humans + number of emulators) x (number of detectors), where each element is a mathematical answer to 'is this a human player'.

      Step 3:
      Repeat and take the average score. The Detector that was right the most wins.

      Step 4:
      The emulator that fooled the most detectors wins. If there's a tie (for either emulators or detectors) in the 95% confidence interval for the model used to compute scores, then the prize is shared among the tied entries

      Jason
      ProfQuotes [profquotes.com]

  • It's not hard to write a "gamer emulator" (Score:5, Funny)

    by JessLeah ( 625838 ) on Tuesday January 14, 2003 @10:12PM (#5085073)
    Just make it run around shooting stuff and saying things like "lol u camping fagot!!!!" ;)

    Oh, and "my new vidcadr r0x ur world".

  • Wetware is... (Score:5, Informative)

    by httpamphibio.us ( 579491 ) on Tuesday January 14, 2003 @10:12PM (#5085074)
    According to The Jargon Dictionary [astrian.net] wetware [astrian.net] is:

    wetware /wet'weir/ n. [prob. from the novels of Rudy Rucker] 1. The human nervous system, as opposed to computer hardware or software. "Wetware has 7 plus or minus 2 temporary registers." 2. Human beings (programmers, operators, administrators) attached to a computer system, as opposed to the system's hardware or software. See liveware [astrian.net], meatware [astrian.net].

    I didn't know what it meant... figured other people may not either.
    • Wetware is a software written by WetCat.
      Or it's just a packaged swimsuits.
      Can a robot tell the packaged swimsuits from
      the packaged computer games and software in
      the shelves of Wal-Mart?
    • sad /sad/ adj. 1. Bereft of happiness. 2. Human beings (programmers, operators, administrators) so attached to a computer system that their hardware will never encounter any wetware.
    • The way to differentiate between hardware,wetware and software is thow'em all out the window of a tall building.

      The hardware goes "CRUNCH!", the wetware "SPLAT!" and the software doesn't.

  • get it right (Score:1, Informative)

    by philtulju ( 552080 )
    its "caltech" NOT "Cal Tech"

  • uh oh (Score:1)

    by RyLaN ( 608672 )
    So I guess that if you can't understand their algebra in the 2nd part of the explanation, you aren't elible to join..maybe some /.'er could translate that for those of us who haven't graduated grade school yet?
  • Hang on..so the TT wants me to write something that will play a very limited game in a very limited way? Is that it?

    Sounds pretty damn easy to me!

  • No Hacking Around! (Score:3, Interesting)

    by core plexus ( 599119 ) on Tuesday January 14, 2003 @10:14PM (#5085096) Homepage
    From the description: " The goal of this project is to study human behavior in repeated games, and to find and document the best algorithms available. The goal is not to probe for weaknesses in the Linux operating system. Each participant has permissions to read, write and execute in your own directory (and any subdirectories of it.) You do not have permissions to read, write or execute programs in any other directory on the host computer. Any attempt to read or write from directories to which you do not have privileges will be considered an act of bad faith, and your algorithm will be disqualified from further competition in the tournament. http://turing.ssel.caltech.edu/node19.html

    Well! Never mind, then.

    Fight with computer brings SWAT team [xnewswire.com]

  • Play like a human? (Score:5, Funny)

    by gpinzone ( 531794 ) on Tuesday January 14, 2003 @10:16PM (#5085105) Homepage Journal
    ...if you can program an emulator that will play games like a human...

    What? You mean make a bot to miss every shot using the railgun in Quake 3? I think I can whip up some AI for ya!
  • It's "Caltech", no intercaps, not "Cal Tech", or "CalTech" or whatever other odd spelling that the press came up with last week because no-one pays attention. "Caltech."

    Sorry; it's a relative minor gripe from an ex-Lloydie...

  • First Entry (Score:5, Funny)

    by shoemakc ( 448730 ) on Tuesday January 14, 2003 @10:21PM (#5085121) Homepage
    "The Turing Tournament is a two sided tournament designed to find, on the one hand, the best computer programs to mimic human behavior"

    humanator2()
    {
    while (sex=="false")
    for (0:ii:4294967296) {
    if (ii mod 100!=0) {
    call dwnld_porn(kiddie);
    else
    call mstrbte();
    end;
    ii++;
    }
    }

    Note the function requires no parameters...and eventually self-destructs :-)

    -Chris

  • Interesting.. (Score:5, Interesting)

    by metlin ( 258108 ) on Tuesday January 14, 2003 @10:23PM (#5085131) Journal
    From the site:

    ...
    An emulator is computer program that takes as input a stage game file, and gives as output a dataset file. ...
    The input file is in the file game.nfg, and the output file should be written in the file dataset.txt, both of which should reside in the same directory as the executable program. Note that the file game.nfg will be written into each emulator's directory by the tournament program prior to running your emulator. ...

    This is exactly the way ACM ICPC contests are conducted, except that if the systems crash for any reason, you're not given extra time to make up for it :-(

    And sadly, from the site:

    Languages supported:


    The computer program that you submit (for either an emulator or a detector) must be written in a combination of one or more of the following languages:

    C or C++
    java
    Perl
    Mathematica
    Gambit GCL

    I would have expected them to atleast add shell-scripting to this - very useful under such conditions to do some Q&D work, or would be taken for granted that since the shell can be a part of the OS, you are free to use it?

    Also, would have been nice if they'd added Python to the list, and more importantly Forth (yes, despite what you've heard, Forth is indeed useful, just look at Arthur T Murray's Mind Project [sourceforge.net]).

    • In the ICPC, the set of available languages is restricted because the participants are using secured computers (to keep them from getting on the Net and causing trouble on the LAN). You'd think it wouldn't be a problem if participants could submit a binary compiled at home. However the whole point of this contest is for Caltech to see what ideas everyone else has come up with -- so they want to make sure they can read your source code.

    • writeln 'What?!? No Pascal? They''re biased!'
    • Perl is on the list. What commands would you want to use in a shell script that aren't fairly easy to implement in Perl?
    • I would have expected them to at least add shell-scripting to this
      If you take a look at this section [caltech.edu] you will see that they are pretty flexible about language:
      We intend to eventually allow submission of entries in other languages besides those mentioned above. If you prefer to write your algorithm in some other language or software, please contact the Turing group via email at ...
      Also they mention the possibility of using a shell script as a wrapper.
    • Ummm.. Scheme, LISP? Where are they. Guess I'll just write an interpreter in C or Java and have it load a Scheme source file.

      • And they missed ML too.

        But regarding writing interpreters in C or Java for Scheme src file, wouldn't that be easier done in something like Perl?

        I was thinking along the same lines too, but isn't there a proof of concept of this, because I remember reading about it somewhere? I do know that MIT has a LISP interpreter in Perl, though.

        I guess there is a Scheme interpreter in Java (Jscheme) with an existing code base that could be used. Any idea if there's one in Perl/Python?
        • Writing a Scheme interpreter in any language is pretty easy, it's a very simple language to implement. I love Perl (and I'm going to get flamed for this) but I wouldn't use it as an interpreter for Scheme. It's just not the right tool for the job. I used to write everything in Perl unless I needed performance at which point I moved to C, kinda like the write in C then switch to assembler for performance that I did years ago. But Perl tends to get unwieldly at certain project scopes, for which I'm partially to blame but the language doesn't lend itself well to scaling.


          I'm guessing somewhere there is a Scheme interpreter for both Perl and Python, and if there isn't (disregarding my previous paragraph) I may write one, because it's a fun project. I've never searched for them, but there are probably lex and yacc like tools for Perl and Python which would simplify the building of an interpreter. Of course if they don't exist yet... they should and that's another interesting project.

  • others have felt that way before

  • Turing Test? (Score:5, Interesting)

    by $carab ( 464226 ) on Tuesday January 14, 2003 @10:24PM (#5085140) Journal
    As someone who spent a lot of time working with an actual Turing Test bot, I'd just like to say that the term "Turing Test" really only applies to a "conversation" between computers and people, not emulating behavior in games, etc. I mean, when Turing wrote Computing Machinery and Intelligence [loebner.net], this isn't really what he envisioned to be a Turing Test.

    From what I could gather, this is a lot closer to a programming tournament rather than a Turing Test...

    • Re:Turing Test? (Score:3, Interesting)

      by Vagary ( 21383 )

      As someone who spends more time reading research papers than working with an actual Turing Test bot, I'd just like to point out that the academic community has abstracted the term "Turing Test" into something a little more useful:

      A Turing Test is a means for judging the humanness of the behaviour of a software system. The test consists of giving similar input to both a human and an implementation of the software and then comparing or subjectively judging the output. This is often done in an interactive fashion.

      Idealised, a Turing Test consists of a human and a program receiving a bitstream and sending a bitstream back. In most examples, this bitstream consists of text in a natural language and the input and output are expected to occur interactively. However it seems likely that intelligent aliens would fail this if they didn't know the language or could not respond at a speed acceptable to the judge. Therefore your traditional definition of "Turing Test" is seen to be arbitrary.

      And no, citations are not available upon request.


      • > As someone who spends more time reading research papers than working with an actual Turing Test bot, I'd just like to point out that the academic community has abstracted the term "Turing Test" into something a little more useful...

        Games would actually make a great Turing test. I.e., monitor a networked game and try to tell which players are humans and which are artificial agents.

        For most games that would be much easier than the unrestricted conversation test, but maybe that's where we need to go to get to first base with AI.

    • isn't a game a simple form of communitation.
      • No, it isn't.
        • Blox, yes it is. It depends how you play the game. It's certainly more of a language then a Monkey point at symbols.

          The challenge appears to be play the game like a human. Well when most humans play a game they use a kind of language not just logic,

          player 1 makes a move
          player 2 looks at the game and try to assess what player 1's upto, and what move to make.
          repeat.

          In language, say an IRC chat,
          Person 1 says something,
          Person 2 looks at what person 1 has said and try to assess what it's all about, where person 1 is going with the conversation etc....

          It becomes even more like a language if the computer is only given basic rules of the game and the programme has to work out how to play.
          It's like giving an AI bot basic grammar and the bot having to work out what the words mean.

          If you don't play the game like you use language then the detector bot should pick you out as a computer.

  • I never understood this. Why would I want a computer that ridicules my hair, dress, and generally pathetic life? If I wanted that, I'd just get married!

  • Game Undefined (Score:2, Informative)

    by Vagary ( 21383 )

    The rules are written in a very obscure minimalist fashion, so it took me a while to figure this out but: the game has not been defined! Your program is to get an input file and process it in a manner similar to a human. Currently the website is lacking examples of human output. Therefore, from an Information Theory perspective, we know absolutely nothing about the game.

    Now, since I wasted my time figuring this out, I also decifered the instructions: basically the pool consists of a bunch of humans and "emulators" (programs). Each one is given a set of input files that they are supposed to transform into output files. Then the set of output files is run through a detector (human or machine?) that gives the participant some score. You win if your score is most like the humans' scores.

    • completely wrong (Score:4, Informative)

      by jbellis ( 142590 ) <jonathan@carDEBI ... com minus distro> on Tuesday January 14, 2003 @10:47PM (#5085258) Homepage
      you must have missed this node [caltech.edu] and possibly this one as well [caltech.edu]

    • Re:Game Undefined (Score:5, Informative)

      by CommieOverlord ( 234015 ) on Tuesday January 14, 2003 @10:52PM (#5085280)
      The game has been defined. It's fairly classic payoff matrix used in game theory. Look up the Prisoner's Dilemma (http://serendip.brynmawr.edu/bb/pd.html) problem. The goal of the game is maximize your outcome (while minimizing the opponent's).

      • interesting, i created a program to evolve a strategy (EA) for that very game last semester. It's not just used in game theory, it has applications in psychology and political science, too. oddly enough, the best general strategy against unknown players is one of the simplest - Tit-For-Tat. Defect when the opponent defects and cooperate when your opponent cooperates. now i am wondering if this kind of knowledge is really worth $10,000...seems more like a class project to me. :)

        --paul

        • Or you alternatively say that game theory itself has uses in other fields of study.

          The program is a bit more complex than the standard two players/two options choices. If each player has 10 options, then does the Tit-for-tat strategy still work? If the payoffs aren't symmetical does Tit-for-tat still work? Seems a bit more complex, but probably not overly so.

    • I think you're incorrect, the game *has* been defined, albeit with a a set of parameters which *you* will have to optimize

      This is more of a resource management problem of optimizing your dataset while making sure that the opponent's is at a pessimum [google.com] the other. This is infact used in manufacturing and related areas too.

      Rather, I guess you mean that the *means* of doing so have not been defined.
    • The class of games is extremely well defined. But the specific pay-off matrices will be input to the emulator. That is, an emulator will have to read the pay-off matrix and decide how to play it. Also note that many games of this nature will be played.

      There is a good reason for this. If the game (or a small finite set of games) were pre-defined, it would be easy to have a bunch of human subjects play it and then have the emulator regurgitate such a "book". Most entries, I suspect, will be from people or teams who are familiar with studies of how real people do play such games.

    • Obviously I didn't read closely enough. Thanks to everyone who covered my err.

  • A flash version of the game is available here [flasharcade.com] for those too lazy to download.

    Enjoy!

  • The Turing Test (Score:2, Informative)

    by Rally Ball ( 641238 )
    The Turing Test was developed by Alan Turing as one of his "free time endeavors." (I swear, that man had too much time on his hands.) Either way, the Turing Test has been the unsurpassed test for true AI and a human-wetware algorithm. The Turing Test dictates that only when a computer is fully capable of handling a conversation with a human, being able to respond to questions in logical, grammatically correct formats AND being able to learn new slangs and vocabularies - that is true AI (according to Turing). I think this CalTech game is about the same - it's not merely making a computer make mistakes and saying human-like taunts. It's truly.. being human. Those once-in-a-while lucky shots, beginner's luck, fresh leg advantages, the works. That's what they're looking for. Besides, it's CalTech. I think they deserve a little more credit than just merely modifying existing AI. =)

  • More Basic (Score:5, Funny)

    by shoemakc ( 448730 ) on Tuesday January 14, 2003 @10:34PM (#5085197) Homepage
    REM HUMAN V2.3

    10 BUILD STUFF
    20 WANT MORE STUFF
    30 BUILD MORE STUFF
    40 WANT STUFF YOU CANT HAVE
    50 BUILD WEAPONS TO TAKE STUFF FROM WHO DOES HAVE
    60 GAIN ENEMIES
    70 BUILD BIGGER WEAPONS FOR DEFENSE
    80 BUILD SUPER WEAPON
    90 DESTROY SELVES
    99 GOTO 10
    • Christ! You couldnt even bother to put it in C/C++ or a scripting language? Hell, you could have put it in VB even.
    • REM WARCRAFT V2.3

      10 BUILD STUFF
      20 WANT MORE STUFF
      30 BUILD MORE STUFF
      40 WANT STUFF YOU CANT HAVE
      50 BUILD WEAPONS TO TAKE STUFF FROM WHO DOES HAVE
      60 GAIN ENEMIES
      70 BUILD BIGGER WEAPONS FOR DEFENSE
      80 BUILD SUPER WEAPON
      90 DESTROY SELVES
      99 GOTO 10

  • AI tournament without AI languages? (Score:5, Interesting)

    by jtdubs ( 61885 ) on Tuesday January 14, 2003 @10:40PM (#5085218)
    Amazing. They are having an AI tournament, and their supported language list includes C, C++, Java, Perl, Mathematica, and something called the Gambit Command Language.

    Where the hell are the good AI languages? Functional languages? Lisp? Scheme? Caml? SML? Hell, I'd settle for Python.

    Justin Dubs


    • They are having an AI tournament, and their supported language list includes C, C++, Java, Perl, Mathematica, and something called the Gambit Command Language.

      They're having an AI tournament on something that's more related to Game Theory and which is why GCL has been mentioned.

      GCL is a HLL that's used for testing game theory related approaches. It supports a lot of important factors in game theory related operations, like vectorization and form representaion switching.

      Read this Caltech site [caltech.edu] for more on GCL.

      GCL may not be very well known outside the AI/GT areas, since its used more in a purely CS research oriented environment. I think it started out as a series of C++ libraries for GT related stuff.


    • > Amazing. They are having an AI tournament, and their supported language list includes C, C++, Java, Perl, Mathematica, and something called the Gambit Command Language. Where the hell are the good AI languages? Functional languages? Lisp? Scheme? Caml? SML? Hell, I'd settle for Python.

      Damn! I'm 30 million lines into my HAL 9000 emulator, but I wrote it in COBOL.

    • They are having an AI tournament, and their supported language list includes C, C++, Java, Perl, Mathematica, and something called the Gambit Command Language.

      Which one isn't Turing Complete? I swear, that you got moderated up to 5 with anything other than Funny is a crying shame.

      • Congrajulations! You are now officially an idiot!

        Yes, oh wise one, they are all turing complete. So is Basic. So is assembler. So is binary. So are Fortran and Cobol. But I wouldn't want to write AI in them.

        If you do, than that's your business, but you won't find many people agreeing with you.

        Languages have domains. C is good at system programming. Perl is good at manipulating strings. Lisp is good at AI. You can use Perl for systems programming and Lisp to manipulate strings, but your code will be a lot longer and harder write.

        Or, can I assume that you know more computer languages than I do (28), have been programming longer than I have (10 years), or have some other quality which would make your opinion anything other than dumb.

        Justin Dubs

        • Congrajulations! You are now officially an idiot!

          Why, because you think I'm an idiot? I somehow think my ego can withstand that level of attack. The only thing that is clear here is that you have no idea what AI is about.

          Languages have domains.

          If you actually believe that, you have no chance of ever writing a single program that even approaches Eliza-level intelligence. You are confusing the quality of human understanding and organization that a language may provide with the algorithms that are use to organize information internally by the program. And whatever (in)efficiencies and understanding might be abstracted into a higher level language for humans, it all gets run as a machine language representation. So next time, before going off so cocksure, try to gain an understanding of the subject matter you're talking about instead trying to appear like an authority you obviously are not.

          I stand by my having said you should only have been modded up as Funny. I know I sure chuckle more with each post you make.

          • Haha... You get off on trying to condescend to kids half your age and yet twice as smart as you. Good luck with that, Doctor. Haha.

            Mine delusions aquainted
            Bubbles erotica
            Plutonium wedding rings
            Icicle stretchings
            Bicycle shoe-strings
            One flag, flaggy but one
            Painting the paintings of the alive
            I-E-A-I-A-I-O

  • It's Caltech now. (Score:4, Informative)

    by toybuilder ( 161045 ) on Tuesday January 14, 2003 @10:41PM (#5085230)
    Page sucks!

    Oh, er, hmm. Sorry about that.

    As part of a "branding" attempt after around WWII, California Institute of Technology refers to itself as "Caltech", not "Cal Tech".

    See this Caltech Institute Archive [caltech.edu].

  • OK I could be wrong, but,,, (Score:4, Insightful)

    by SuperCal ( 549671 ) on Tuesday January 14, 2003 @10:41PM (#5085234) Homepage
    Its been awhile sence I read about the subject, but isn't the Turing test just putting people in front of a terminal to talk to either a real person or a AI, and then asking which is the real person. When the same number of testers chose the AI as the real person, then the AI passes the test. Sence when did the Turing name apply to every AI competion? Am I wrong? This isn't a flame just a question about definitions...

    • You're right. In a puritan sense, strictly only machine-human conversations constitute this element.

      But that's open to debate, since Turing also mentions about systematic storage and processing of information based on pre-determined or arrivable patterns, for similar functions.

      Since this competition does use Game Theory elements to do something along those lines, it would be correct from that perspective to call this a Turing Test.
  • Was this post written by wetware or hardware?

    In all seriousness, the Turing test is really old hat. We've had Eliza the shrink for years, hell, I remember in a CS class a story about an app that emulated a paranoid schitzophrenic(sp) that talked with Eliza.

    We've passed the test. It doesn't really mean much as far as AI goes, what's the point? Emulation is NOT simulation. A parrot can emulate a human, that doesnt mean he thinks like one.
    • If you'd read the article (even a little of it) you'd realize that the competition is not a Turing test at all. Unless, of course, you don't know what a Turing test is, in which case you certainly shouldn't be calling the concept "old hat".

      old hat adj.
      Behind the times; old-fashioned: Last year's styles will be old hat soon.
      Overused; trite: That prank is old hat.


      So -- exactly how is the Turing test "old hat" in any way? Being around a long time does not necessarily cause something to be less compelling or lose its usefulness (i.e. Fermat's Last Theorem).

      Is it behind the times? If so, what's the modern AI test? C'mon, a better one please.

      Is it trite? (Lemme help ya, just in case:)

      trite adj. triter, tritest
      Lacking power to evoke interest through overuse or repetition; hackneyed.
      Archaic. Frayed or worn out by use.


      If you think either of those apply, then you truly do not understand what a Turing test is.
    • Comment removed based on user account deletion
    • We've passed the test

      Sorry I missed that comment -- I really am. Exactly when did a computer "pass the [Turing] test?". If you meant "we", as in us humans, you don't understand what a Turing test is.
  • I mean, maybe if you had tens of millions of dollars to analyze how humans do specific tasks compared to computers you might be able to code something up, but just guessing? I don't believe anyone can come up with solutions to this without doing huge amounts of research.

    It's one thing to try to write a program that does things as well as humans, it's a whole other things like humans. And it's impossible without an operative definition of 'humanness'

    Especially given that there is such a huge range of how 'well' people do things. I mean, some people are idiots, and some are geniuses. How can anyone write a program that can tell the difference between Deep Fritz and a grandmaster, and a program making lots of intentional errors and someone who doesn't know chess? And every class of player in between?
    • if you had tens of millions of dollars to analyze how humans do specific tasks compared to computers you might be able to code something up, but just guessing? I don't believe anyone can come up with solutions to this without doing huge amounts of research.
      You are, of course, right. But what you don't know is that there has been a huge amount of research on how people behave in such games. These sorts of games have been used to test models of how people think of fairness, how much people prefer absolute outcomes versus relative outcomes (eg, would rather win $5 if the other player gets $1 than win $7 if the other player $9), and, of course, the imfamous prisoner's dilemma.

      So, for many classes of games that come up, particular researchers will have serious ideas about how people behave (and may have conducted thier own experiments).

      Someone who shared the most recent Nobel Prize in Economics was for his work on experimental game theory. I expect that he will be submitting something.

  • After reading the specs really quick, here comes a simplified summary of how the games work. Don't sue me if this is wrong, I did not look too carefully and it has been more than a year since I left Caltech.

    Basically, the two players get to choose numbers with different pay-offs for themselves and the other player.

    If you are short-sighted, it is easy to write an 'optimal' program, you always pick the max payoff for yourself.

    However, you can also cooperate so that you maximize the average payoff for both players. Hopefully your opponent will realize this and also start picking numbers in this way, if not it is probably wise to go back and be short-sighted and selfish.

    Now, the real object of this is to make (or detect) human-like opponents. I would guess that real human players are very irrational, for example they can get pissed if the other player is too selfish and then demand revenge, even though it does not maximize their own payoff.

    Tor
  • Maybe it's just me, but "Before you get too excited" and "gory details" don't really match up with the heading "Turing Test Competition At CalTech".
  • Things that should have never been allowed to live: GCC "2.96" (really a CVS snapshot done by Redhat as they wanted something more up-to-date than the lovely 2.95.3). This means, that my code (which will probably be running using -Wall -Werror -ansi -pedantic), may/may not compile, depending on such things as the phases of the moon.

    WTF is going on? Why are they using the broken gcc? Can the Caltech ppl who set this pass a Turing Test? I doubt it....

  • What humans are you talking about here? (Score:4, Funny)

    by rufusdufus ( 450462 ) on Tuesday January 14, 2003 @11:12PM (#5085367)
    The following is an example game file your program is supposed to output a dataset for that is "most human". I give you:

    NFG 1 R "game1" { "1" "2" } { 2 2 }

    21 3 3 5 3 5 5 3
    What is the most human response? Anyone? Anyone?

  • Where do they get the humans? (Score:5, Funny)

    by forii ( 49445 ) on Tuesday January 14, 2003 @11:28PM (#5085423)
    The Turing tournament is a two sided tournament designed to find the best computer programs to mimic human behavior.


    I don't think Caltech is the best place to determine what is "human" behavior.



    Life at Caltech. [cripplingdepression.com]



    Also, notice that this contest is being held by the "Division of Humanities and Social Sciences".

  • Pseudoscience! I call bullshit! (Score:5, Interesting)

    by rufusdufus ( 450462 ) on Wednesday January 15, 2003 @12:26AM (#5085676)
    This 'contest' reminds of the scene in Ghostbusters where Bill Murray manipulates pre-cog cards to get women. Really, if a ESP buff were to present something like this to the Amazing Randi he would not accept it on the grounds that it was too easy to manipulate.

    First and foremost, there is a large sum of money being bandied around. The participants are incented to win by monetary payouts, and two payouts of $10,000 dollars are at stake as well. When games with this high of stakes are being played, great caution is generally used by the house.

    But look at the rules of the game...there basically are none. Participants are identified by e-mail address; no rule is specified about the number of entries per person. Also, no rule is specified about collusion between entrants (detectors and emulators). It doesn't take an einstein to figure out how to bias the results of the experient by making enough colluding entries. It is funny this, given that the games themselves are *about* collusion. Its a joke.

    Next, notice that there really isnt any way for anybody to tell if the results of the experiement are meaningful, or if they have been manipulated. Its not based on a falsafible proposition and is not scientific. Its no different from any method used by psychic hoaxers of the past. The creators of the 'contest' can manipulate the data, and direct the winnings to their confederates.

    Finally, the bizarre nature of the contest should raise some flags. They are giving out $10,000 for a program that would have been hacked together in a couple of months at most. What kind of value could you expect from a contest like this?
    • Comment removed based on user account deletion
      • I am a retired software engineer.

        If you believe this contest is really about marketing CalTech rather than a scientific approach to sociology or intelligence, I won't argue.

        Note that my post just one way to approach the flaws riddling this contest...

  • Here's what this is *really* about (Score:5, Informative)

    by Cedric ( 3316 ) on Wednesday January 15, 2003 @12:46AM (#5085743) Homepage
    As a recent alum (2002), I can shed some light on some things.

    Why is this in the Deparment of Humanities and Social Sciences?

    From the URL, this Tournament is being run by the Social Sciences Experimental Laboratory (SSEL), not the CS department. The SSEL has been one of the leaders in experimental economics research (read: actually testing all those crazy theories you hear in economics classes).

    Why is there money involved?

    All experiments by the SSEL involve money. As an undergraduate, I participated in many experiments, mostly involving trading "commodities" in simple (and sometimes not-so-simple) markets. We were paid based on our performance. If I had an off night, I got paid $5 for 2 hours of the experiment. If I had a good night, I could make upwards of $80. Yes folks, this is real money we're talking about here. Since the point is to test people's economic thinking, you must make your decisions based on a real outcome, otherwise the data gathered is invalid.

    Why then are they doing this test?

    I don't work for the SSEL (and never have), but here's why I think they're doing this: Since they're interested in not only individual human behavior, but also how individuals interact and make choices based on the actions of other individuals, it would be useful to design a computer program that mimics other human's behavior. If other humans think this program acts like a human, then you can do two things: you can take the specifications of the computer program and figure out what qualities of the program humans have. As well, you can then replace humans with the computer program in real experiments (this not only allows you to test the limits of the program, but also to save money :-) )

    And as an alum (who was not too fond of his time there, but still feels compelled to defend Caltech), it's *Caltech*, not *Cal Tech* or *Cal-Tech* (but if you're feeling lazy, *caltech* is all right too).

    nak

  • One Game (Score:5, Funny)

    by The_Shadows ( 255371 ) <thelureofshadows ... minus physicist> on Wednesday January 15, 2003 @01:24AM (#5085852) Homepage
    The only game I want to play is Global Thermonuclear War.
  • Cal Tech wants to know if you can program an emulator that will play games like a human,

    That would be trivial for a MMORPG. Just write a program to make the character stand around at popular spots, like vendors or quest dispensers, casting doubt on the manhood of passerby, and it will be indistinguishable from your typical 13 year old gamer. It could even be done with an existing program like Eliza, if its conversation database were tweaked to generate juvenile insults.
  • Caltech is a private university and not part of the UC system. To refer to it as "Cal Tech" or "CalTech" is not only wrong and misleading, but does a disservice to the people associated with that fine institution.

    That said, I think the stuff happening there is very cool.

Slashdot Top Deals

Remember to say hello to your bank teller.

Close