« Health care and them annie-mules, update | Main | The Increased Competitiveness of the US Economy »
What does the Turing test really mean?
That is a new paper of mine, co-authored with Michelle Dawson. There is much more to Turing's classic essay than meets the eye. The famous "test" is not a standard for distinguishing human from machine intelligence but rather one step in an argument showing that such a distinction is not as important as we might think. Turing cleverly shows why the supposed test is misleading and the real question is how to educate both children and machines, not how to distinguish them. The summary statement of our paper is as follows:
...a potent and indeed subversive perspective in the paper has been underemphasized. Some of the message of Turing’s paper is encouraging us to take a broader perspective on intelligence and some of his points are ethical in nature. Turing’s paper is about the possibility of unusual forms of intelligence, our inability to recognize those intelligences, and the limitations of indistinguishability as a standard for defining intelligence. “Inability to imitate does not rule out intelligence” is an alternative way of reading many parts of his argument. Turing was issuing the warning that we should not dismiss or persecute entities which we cannot easily categorize or understand.
If you read Turing's essay closely, you will find many underrated passages of interest, especially when read in light of his homosexuality (and also possible autism). Here is the closing bit from our paper:
It is possible that Turing conceived of his imitation test precisely because he had so much difficulty “passing” and communicating himself. In social settings these facts were seen as disabilities but in the longer term they helped Turing produce this brilliant essay.
One brute fact is that a lot of human beings could not, themselves, pass a Turing test. Could you?
Addendum: Here is my previous post, Toward a Theory of Raivo Pommer-Eesti.
Posted by Tyler Cowen on July 15, 2009 at 07:18 AM in Education | Permalink
Comments
uggzcl: Would you say Mr. Pickwick reminded you of Christmas?
Posted by: Paul Zrimsek at Jul 15, 2009 7:52:46 AM
"One brute fact is that a lot of human beings could not, themselves, pass a Turing test. Could you?"
I don't think I buy that -- unless 'a lot' means those humans with severe brain injuries or language deficiencies. But otherwise, I don't think that there are 'a lot' of humans who would be mistaken for a computer more often than not.
The Turing test has always seemed to me to be the wrong sort of tool (or wrong sort of thought experiment). That is we've had examples of extremely simple artificial systems that can fool some of the people some of the time (e.g. ELIZA) but are clearly not intelligent. And we have various forms of intelligent artificial systems that couldn't begin to take, let alone pass, a Turing test (the kinds of autonomous vehicles that participate in the defense department's DARPA challenge events). The TT doesn't even do much for us in determining moral claims. So there is little doubt that chess playing systems that can beat the world's best humans possess a sophisticated (specialized) form of intelligence, but nobody feels these systems also possess inalienable rights not to be damaged or destroyed. But there are many TT incapable creatures without language capabilities that do possess such rights (one of which is curled up on the floor at my feet here).
Posted by: Slocum at Jul 15, 2009 7:54:43 AM
"The famous "test" is not a standard for distinguishing human from machine intelligence but rather one step in an argument showing that such a distinction is not as important as we might think. "
Sorry, but I do think this mixes truth with rubbish. Turing understands the test isn't a standard for distinguishing - after all, he thinks a machine will be able to pass his "imitation game", so there's no distinguishing there, right? So your first conjunct is correct; it's not a standard for distinguishing and Turing never suggested it as such.
But the second conjunct - the test is "one step in an argument showing that such a distinction is not as important as we might think" - is just rubbish. It's no such thing. Turing would like to answer the question, "Can machines think?" but realizes that the question can only be answered if it is clear what one means by "think". He doesn't want to get bogged down in semantics, so he proposes an alternate question, using his famous test, "which is closely related to" the question "Can machines think?" and "which is expressed in relatively unambiguous words".
Posted by: a at Jul 15, 2009 7:58:40 AM
Turing didn't expect many people to pass the test back then either. It's a man/woman test, not a computer/human test. A coin toss could get a 50% on the Turing test.
Posted by: Phil at Jul 15, 2009 8:24:23 AM
Surely the odds of me, or anyone else, passing a Turing test is not independent of the qualities of the administrator?
If we assume a very competent test administrator, then yes, I'd pass. If we assume a lazy test administrator who just picks at random, I'd have a 50% chance. If we assume an incompetent test administrator I might have a lower than 50% chance (judging by people on the Internet who have occasionally said "You only say that because you're male.")
Posted by: Tracy W at Jul 15, 2009 8:40:24 AM
The "Juicy" spam by uggzcl that begins the comment section (and which will likely be gone by the time you read this) is surprisingly apropos to the topic of Turing tests and AI in general. The arms race between spammers and anti-spammers will probably do more to further the field of AI than any academic research, for the simple reason that it will attract far more funding.
The holy grail for spammers would be a chatbot that could pass the Turing test and do word-of-mouth spamming in far more subtle and effective way than is currently possible; the holy grail for anti-spammers would be AI that could parse a message and categorize it as spam or non-spam as correctly as a human could, without false positives.
Posted by: anonymous at Jul 15, 2009 8:56:03 AM
PS, the above condensed into a bite-size meme:
"The Russian mafia: midwife to the singularity"
Posted by: anonymous at Jul 15, 2009 9:18:41 AM
One brute fact is that a lot of human beings could not, themselves, pass a Turing test. Could you?
I don't see what this means. Do you mean that two humans are being interrogated, by someone who has been told that one is a machine?
I suppose some people would be picked less often as being the machine than others in that case. But I doubt there are many people who would do worse than actual machines.
I always thought the Turing test was obviously a proposal for a sufficient test, not a necessary test.
Posted by: Zamfir at Jul 15, 2009 9:31:36 AM
If the test administrator is trained, then they will almost always correctly choose the human (versus a program attempting to fool the administrator). OTOH, if you take a random person off the street, the program will fool them a reasonable percent of the time.
One thing to note though is that this isn't a very active area of research. It's not considered very useful to make AI act like humans (we have humans to do that for us). So if everyone cared more it's likely we'd be in a much better position.
Posted by: Andy at Jul 15, 2009 10:06:49 AM
Could a machine blog?
Posted by: Unit at Jul 15, 2009 10:06:52 AM
I don't know. Have you ever chatted with the Indians running the Dell help desk? They have human-triggered hot-keyed responses to the customers questions, sort of human-assisted computer conversation, or a computer-assisted human conversation. Either way, I had no feeling I was involved in an intelligent discourse.
Posted by: Andrew at Jul 15, 2009 10:08:52 AM
BTW, is there somewhere any research that people are actually fooled by ELIZA type programs when they are actively told to identify a machine? It seems unlikely to me that even untrained random people from the street would be fooled for long, but who knows.
Posted by: Zamfir at Jul 15, 2009 10:53:15 AM
Dear Mr. Cowen, re your piece on autism: I would have you know that the greatest genius of a currently living writer and dramatist, Peter Handke, is autistic: after doing as thorough a possible [for me] analysis of his capacities as a writer, and of the profound benefits and psychological wounds he acquired intrauterine and in childhood, his social ineptness and the autistic states that his extreme hypersensitivity bequeathes, of which we as sensitive readers can be the beneficiaries, would appear to be a differently arranged sets of brain stemcells, allowing of the growth of certain areas that are generally killed, or anyhow, too "socialized" by socialization processes. Something along those lines, explains the Mozarts, the Bachs the Handkes, the Beethovens, to stick just to the German realm, and that of course does not constitute an explanation, perhaps a setting of the viewing angle. here is a set of links to handke material: LINK OF LYNXES TO MOST HANDKE MATERIAL ON THE WEB:
http://www.handke.scriptmania.com/favorite_links_1.html
HANDKE LINKS + BLOGS
SCRIPTMANIA PROJECT MAIN SITE: http://www.handke.scriptmania.com
and 13 sub-sites
the newest:
http://handke-photo.scriptmania.com/
contains the psychoanalytic monograph
http://www.handkelectures.freeservers.com [the drama lecture]
http://www.van.at/see/mike/index.htm
[dem handke auf die schliche/ prosa, a book of mine about Handke]
http://begleitschreiben.twoday.net/topics/Peter+Handke/
http://handke-discussion.blogspot.com/
[the American Scholar caused controversy about Handke, reviews, detailed of Coury/ Pilipp's THE WORKS OF PETER HANDKE, the psycho-biological monograph]
with three photo albums, to wit:
http://picasaweb.google.com/mikerol/HANDKE3ONLINE#
>
http://picasaweb.google.com/mikerol/HANDKE2ONLINE#
http://picasaweb.google.com/mikerol/POSTED?authkey=YeKkFSE3-Js#
http://www.handke-trivia.blogspot.com
http://www.artscritic.blogspot.com
[some handke material, too, the Milosevic controversy summarized]
MICHAEL ROLOFF
Member Seattle Psychoanalytic Institute and Society
this LYNX will LEAP you to all my HANDKE project sites and BLOGS:
http://www.roloff.freehosting.net/index.html
"MAY THE FOGGY DEW BEDIAMONDIZE YOUR HOOSPRINGS!" {J. Joyce}
"Sryde Lyde Myde Vorworde Vorhorde Vorborde" [von Alvensleben]
Posted by: MICHAEL ROLOFF at Jul 15, 2009 11:00:56 AM
Way off topic:
Please stop putting two spaces between your sentences. I understand that this is probably just an artifact left over from your days of using a mechanical typewriter, but I have to assume you're using a computer with variable-width fonts by now. Unless you're using a fixed-width font, sentences should *never* be separated by two spaces. HTML itself doesn't even support this, as you'll notice by the kludge that is employed to add your double-spacebar formatting:
From the first sentence (source code view):
...authored with Michelle Dawson. There...
Your blogging software is nice enough to respect your wishes and implements the kludge to override the correct formatting, but it really shouldn't. Sorry for the rant.
Posted by: OCD at Jul 15, 2009 11:26:18 AM
MICHAEL ROLOFF and OCD pass.
Posted by: Andrew at Jul 15, 2009 11:38:29 AM
OCD,
Double spaces after sentences increase the amount of white space in a passage and often do so without noticeably increasing the length. This is good for enticing people to read, since many readers skim rather than pay meticulous detail to every character.
Looking less "squished," in general, increases the number of people who will read a particular post while bothering only a small amount of the meticulous readers with certain viewpoints on such double spacing. Since character width is often variable anyway, it is in reality a matter of preference whether to double space, so negative consequences to double spaces after sentences are further limited only to a subset of those meticulous readers.
Cost/benefit is likely in favor of double spacing after sentences.
Posted by: David Dittell at Jul 15, 2009 11:56:28 AM
"Unless you're using a fixed-width font, sentences should *never* be separated by two spaces."
Why?
"HTML itself doesn't even support this, ..."
Then to Hell with HTML.
Posted by: a at Jul 15, 2009 12:08:36 PM
Awesome Andrew! I hate it when people declare a thread winner, but, you win, man.
Posted by: kebko at Jul 15, 2009 1:20:27 PM
Back to Turing - However amazing his work was, practical application to AI was kind of lost decades ago. There are lots of areas where we cannot distinguish between a human and a machine. Using the Indian Call Center example (true for any call center, actually) - the only way we know we are speaking to a human is the interface. The words are scripted and come out exactly the same - human or machine.
Another example - hand drawn animation vs. computer generated.
As suggested above - the criteria for good AI has changed.
Posted by: mid atlantic geography dude at Jul 15, 2009 1:41:17 PM
it's a sufficient test. anything that could truly pass a turing test would have to be intelligent. calling something of that sophistication, complexity, and adaptability anything other than "intelligent" just makes the word meaningless (or turns it into a religious argument about having a soul or something).
of course when I say "truly" I'm talking about an idealized thought-experiment that basically amounts to "passing" for human. or, more generally, "passing" for intelligent. the point is that there isn't in fact any way to "pass" for intelligent except by being intelligent. and the flip side is that there isn't anything to *being* intelligent but acting intelligent (i.e. no magic organic soul).
Posted by: mike at Jul 15, 2009 1:50:26 PM
Double spaces rule!
Posted by: Doug at Jul 15, 2009 1:53:25 PM
mid atlantic dude:
the reasons the indian call center example (by which I assume we mean it's hard to distinguish between a human manning a call center and a computer just producing keyword-invoked help text in a conversation form) works are (a) the humans are severely restricted in their expression by being non-native speakers and having rigid scripts; and (b) the domain is greatly limited, making it easier for the computer to know all there is to know. so of course they're more likely to pass. I doubt you'd be confused if you asked your call center chatter a couple of small talk quesitons, assuming he answered (which they probably wouldn't).
hand-drawn animation vs. computer generated is interesting but has nothing to do with tests for intelligence. sure, our conception of AI has changed, but the point (the philosophical point) of the test as a thought experiment remains the same. we may not be able to decide whether chess computers and call-center bots are intelligent or not, but the day you have a lovely conversation with a robot about movies and strawberries and sunsets and the baseball game and tell a few jokes you will *know* we've got AI.
Posted by: mike at Jul 15, 2009 2:07:17 PM
And THAT is the Turning test we geeks tend to fail.
Posted by: Right Wing-nut at Jul 15, 2009 3:05:48 PM
haha and that is why the test is sufficient but not necessary!
Posted by: mike at Jul 15, 2009 3:12:49 PM
AI nuts: Who's doing work on applying evolutionary algorithms to AI? Like blind variation and selective retention of various algorithms? Has anybody done this in multiple layers? What's going on in this field right now?
Posted by: Michael F. Martin at Jul 15, 2009 4:06:12 PM
