Turing Test Project: Experiment

What happens when one chatterbot converses with another chatterbot? Do they recognize each other as artificial intelligences?


As you can see, the female chatterbot immediately recognized that the male conversant was a chatterbot. Perhaps the female recognized certain responses, or the answers from the male were too mechanical-sounding, or perhaps the male was inhumanly polite.

Is it that easy? Perhaps not...


Now it’s your turn! Your job is to apply your own Turing Test protocol to the three chatterbots from the Black-box Testing Chatterbots activity:

  1. Eliza
  2. Cleverbot
  3. Program-O

First, download the Turing Test—Practice spreadsheet where you may record the results of your Turing Test experiments.

There are five individual worksheets in the spreadsheet. Each of these sheets describes a suggested strategy for you to try with each chatterbot. Experiment with different questions within each category and see how the chatterbot responds. Try to ask as many questions as possible, and ask each chatterbot the same questions to gauge how they respond differently. Think about the following questions in your analysis:

  • How do the responses differ?
  • Does the order in which questions are asked make a difference?
  • Which questions are more effective for your purpose?
  • Are there useful questioning strategies you have discovered that do not fit in the five pre-defined categories?

Class discussion

  1. What happened when you used memory-dependent language?
  2. What happened when you referred to activities outside of “pure thought/cognition?”
  3. What were the chatterbots’ perceptions of consciousness?
  4. What happened with computationally difficult tasks?
  5. What happened when you elicited reasoning (i.e., asked ‘why’)?
  6. Did you use any strategies beyond these? If so, what were they and how did they perform?
  7. How can you use your knowledge of the experiences gained in this activity to design a better ’bot?
  8. What would make for a better test, accounting for these possible modifications?