Wizard of Oz experiment

In the field of human–computer interaction, a Wizard of Oz experiment is a research experiment in which subjects interact with a computer system that subjects believe to be autonomous, but which is actually being operated or partially operated by an unseen human being.

The phrase Wizard of Oz (originally OZ Paradigm) has come into common usage in the fields of experimental psychology, human factors, ergonomics, linguistics, and usability engineering to describe a testing or iterative design methodology wherein an experimenter (the “wizard”), in a laboratory setting, simulates the behavior of a theoretical intelligent computer application (often by going into another room and intercepting all communications between participant and system). Sometimes this is done with the participant’s a-priori knowledge and sometimes it is a low-level deceit employed to manage the participant’s expectations and encourage natural behaviors.

For example, a test participant may think he or she is communicating with a computer using a speech interface, when the participant’s words are actually being secretly entered into the computer by a person in another room (the “wizard”) and processed as a text stream, rather than as an audio stream. The missing system functionality that the wizard provides may be implemented in later versions of the system (or may even be speculative capabilities that current-day systems do not have), but its precise details are generally considered irrelevant to the study. In testing situations, the goal of such experiments may be to observe the use and effectiveness of a proposed user interface by the test participants, rather than to measure the quality of an entire system.

John F. (“Jeff”) Kelley coined the phrases “Wizard of OZ” and “OZ Paradigm” for this purpose circa 1980 to describe the method he developed during his dissertation work at Johns Hopkins University. (His dissertation advisor was the late professor Alphonse Chapanis, the “Godfather of Human Factors and Engineering Psychology”.) Amusingly enough, in addition to some one-way mirrors and such, there literally was a blackout curtain separating Jeff, as the “Wizard”, from view by the participant during the study.

...
Wikipedia