Tuesday, April 19, 2005

On Saturday April 16, students at Simon’s Rock College in Great Barrington, Massachusetts and Dr. Richard Wallace of the A.L.I.C.E. AI Foundation for their first time tested Alan Turing’s thought-experiment. The Imitation Game, based on the original Turing model for testing the ability of humans to recognize artificial intelligence (AI), was carried out with nearly eighty human and AI participants.

The ‘Original Imitation Game’ is described in Turing’s 1950 paper. A popularized version now dubbed the “Turing Test” involves a judge knowingly interviewing a software program and a human person during a computer chat, and then trying to discern which is which. The Turing Test has been conducted many times as Artificial Intelligence programs developed. However, no study was ever published following the guidelines of the original thought-experiment itself.

The Imitation Game involved playing a “gender guessing game”, wherein two human subjects, a male and a female, communicate via computer chat to the judge. Both the male and the female would try to convince the judge that s/he is female. Turing’s original question was, if a gender guessing game were done with two humans, and then with an AI replacing the male, would the judge be more accurate in guessing who the real female was?

Three students at Simon’s Rock — Cameo Wood, Melissa Leventhal, and Allyson Sgro — wrote a grant to support the experiment, and shepherded the proposal through the Human Research Review Committee under the oversight of Professor Anne O’Dwyer. The experiment was funded by the departments of Natural Science and the department of Social Science at the college.

The experiment utilized a program called A.L.I.C.E., which is designed to hold one end of an interactive conversation. The program was provided by the ALICE Artificial Intelligence Foundation. Dr. Richard Wallace was on hand during the experiment to troubleshoot the AI robot, later gave a lecture about on The Anatomy of A.L.I.C.E. and blogged the event.

Six human subjects from Simon’s Rock composed the human players in the game; the judges were recruited from various non-technical internet communities. Roughly eighty individuals participated in the experiment, which required the organizers to maintain strict secrecy about the experiment until it was concluded. All subjects who participated in the experiment were required to be over 18, not affiliated with the college, and were not allowed any foreknowledge of the use of AI in the experiment. Roughly 70 interviews were conducted over a three hour period last Saturday, via AOL’s Instant Messenger, a messaging tool that allows individuals to write to one another online.

The research team at Simon’s Rock has started to analyze the data they acquired during the experiment and will be writing a paper for publication in the coming months. Inquiries regarding the experiment may be directed to researcher@theguessinggame.net.