Reader Comments

Post a new comment on this article

"Virtual Human" and Participant Questions, etc.

Posted by Jason_R_Finley on 06 Mar 2007 at 01:49 GMT

I found this to be a very interesting paper. However, I have a few unanswered questions.

1) "Virtual Human"
What exactly did the participants think the "virtual human" was? Here's what I could cull from the instructions given to participants: "In this particular study you will be interacting with a female virtual character." (Information_participants_official.pdf)
...
"As part of our research program a virtual character
has learned a set of word-pair associations. The learning is sometimes not exact, but we are testing
a reinforcement learning procedure, to see if the infliction of discomfort motivates her, the virtual
character, to remember the word-pair associations better. ... Of course, the Learner is a virtual person, and so no actual people are being hurt." (furtherinformation.pdf)

I see two clear options:
i) Participants thought the virtual character was an artificial intelligence, not controlled directly by any real humans.
ii) Participants thought the virtual character was an avatar, being controlled by a real human.

The second option seems possible especially considering the now widespread nature of massively multiplayer online computer games, in which people interact with animated characters that are often controlled by other humans somewhere (e.g., Second Life, The Sims Online, Ultima Online, Everquest, World of Warcraft).

The instructions don't make clear which of those options is the case, and participants might interact quite differently with the virtual character if they think a human is controlling it. For example, someone may think that the virtual human is something analogous to a personal avatar, and that the human controlling it may actually be upset by its mistreatment.

Were there any data collected from participants that might shed light on what they thought the situation was with the virtual character?

On a related note, I would have liked to see data on whether participants really did know and believe that no one real was being hurt, especially in light of the paper's repeated appeals to participants' "certain knowledge that no one real was protesting or being hurt by electric shocks." Although this may seem like an obvious assumption, I would hesitate to call such knowledge certain without confirmation from the participants themselves. Furthermore, there is always the possibility that participants believed or suspected something different from what they were told about the situation.


2) Knowledge of the original Milgram studies.
It's not clear whether all participants were asked about possible prior knowledge of the original Milgram studies, or whether it was only in the few noted cases that a participants Milgram knowledge just happened to become salient and was noted. Were all participants assessed for Milgram knowledge, and if so: how, when, and what were the results?


3) Virtual Character animation
How exactly was the virtual character implemented and controlled? Was it simply a movie that was routinely paused until the participant, say, finished reading the next set of words? Or did it exhibit ambient motions that looped while waiting for the next action? Were its actions (e.g., the pre-recorded answers) actually triggered by an A.I. or by an experimenter?

If this study is meant in art as a proof of methodological concept, more details like these would be helpful.


4) Excluded participant
In the "Recruitment" section, it's reported that one participant "was eliminated who had stopped after only 5 shocks." This was a participant in the Visible Condition.

In their (29 Dec 2006) response to Eveaustria's questions, the authors state:
"Another participant withdrew after only 6 shocks, and we were suspicious about that."

Was it 5 shocks or 6 shocks that the participant gave before withdrawing, and which condition ?
This strikes me as an important distinction because it was in response to the 11th question (which should elicit the 6th shock) that the learner first protested, shouting “Stop the experiment!” and responding to the experimenter's statement with “Don't listen to him, I don't want to continue!”
This certainly seems a likely place for participants to first consider withdrawing from the experiment.
At any rate, whether this excluded participant withdrew before or after administering the 6th shock after the learner's outburst, why was this considered grounds for exclusion from data analysis?


Final comments:
Perhaps this paper simply follows a format I am unfamiliar with, but I thought the methods and procedures could have been better organized. I was confused by the fact that some information was included in the subsection "Procedures" under "RESULTS" and other information was not found until another subsection labeled "Procedures" under "MATERIALS AND METHODS." More consolidation would have been helpful.

I was very pleased with the inclusion of such details as "The experiment was conducted over several days during July and August 2005 in the Virtual Reality Laboratory at University College London." And I also appreciated that the results of the statistical tests were spelled out in plain terms (e.g., "The Wilcoxon rank sum test rejects the hypothesis that the two samples are from the same population with P = 4.4×10−4.").

I am glad to see that the paper acknowledged the confounds in the Hidden Condition (not seen, not heard, and not protesting) and considers resolving those as possible future directions.

I found the speculations on other potential applications/extensions of the work to be well stated and thought provoking.

This could prove to be an important line of research for multiple fields and I applaud your work.

~jason

~
Jason R. Finley
Graduate Student, Department of Psychology
Cognitive Division
University of Illinois, Urbana-Champaign

603 East Daniel Street
Champaign, IL 61820

http://www.jasonfinley.co...
~

RE: "Virtual Human" and Participant Questions, etc.

melslater replied to Jason_R_Finley on 13 Mar 2007 at 11:42 GMT

QUERY:
1) "Virtual Human"
What exactly did the participants think the "virtual human" was? Here's what I could cull from the instructions given to participants: "In this particular study you will be interacting with a female virtual character." (Information_participants_official.pdf)
...
"As part of our research program a virtual character
has learned a set of word-pair associations. The learning is sometimes not exact, but we are testing
a reinforcement learning procedure, to see if the infliction of discomfort motivates her, the virtual
character, to remember the word-pair associations better. ... Of course, the Learner is a virtual person, and so no actual people are being hurt." (furtherinformation.pdf)

I see two clear options:
i) Participants thought the virtual character was an artificial intelligence, not controlled directly by any real humans.
ii) Participants thought the virtual character was an avatar, being controlled by a real human.

The second option seems possible especially considering the now widespread nature of massively multiplayer online computer games, in which people interact with animated characters that are often controlled by other humans somewhere (e.g., Second Life, The Sims Online, Ultima Online, Everquest, World of Warcraft).

The instructions don't make clear which of those options is the case, and participants might interact quite differently with the virtual character if they think a human is controlling it. For example, someone may think that the virtual human is something analogous to a personal avatar, and that the human controlling it may actually be upset by its mistreatment.

Were there any data collected from participants that might shed light on what they thought the situation was with the virtual character?

RESPONSE:
It is an interesting point, and of course there is no way to know what people were really thinking about this, if anything. My overwhelming impression though is that they would not have thought that there was a real online player controlling the character. The reasons for believing this for two main reasons:

(a) Our instructions – always referring to the virtual character and no other entity.
(b) This never came up as an issue in the interviews, never any hint that there was a belief that that the virtual character was somehow representing an online player.

Another point is that if some of the participants believed that this virtual character represented a real online person, then it is still interesting that they responded with stress to causing harm to that graphical representation of the online person. Recall that in the “hidden condition” where the character could not be seen or heard, the stress responses were much less pronounced. So it seems to be the “seeing and hearing” that is the critical factor.


QUERY:
On a related note, I would have liked to see data on whether participants really did know and believe that no one real was being hurt, especially in light of the paper's repeated appeals to participants' "certain knowledge that no one real was protesting or being hurt by electric shocks." Although this may seem like an obvious assumption, I would hesitate to call such knowledge certain without confirmation from the participants themselves. Furthermore, there is always the possibility that participants believed or suspected something different from what they were told about the situation.

RESPONSE:
Of course it is possible, but unlikely. In retrospect it would have been good to ask something like “Did you at any time believe that a real person was experiencing the shocks?” Again this never came up in the interviews, and was not reflected in any answers in the follow-up letter.

QUERY:
2) Knowledge of the original Milgram studies.
It's not clear whether all participants were asked about possible prior knowledge of the original Milgram studies, or whether it was only in the few noted cases that a participants Milgram knowledge just happened to become salient and was noted. Were all participants assessed for Milgram knowledge, and if so: how, when, and what were the results?

RESPONSE:
Participants were asked in the interviews, and this is mentioned in the paper in the section called “Early Withdrawal”.

QUERY:
3) Virtual Character animation
How exactly was the virtual character implemented and controlled? Was it simply a movie that was routinely paused until the participant, say, finished reading the next set of words? Or did it exhibit ambient motions that looped while waiting for the next action? Were its actions (e.g., the pre-recorded answers) actually triggered by an A.I. or by an experimenter?

RESPONSE:
The virtual character was programmed with a set of predefined animations. The remote operator only controlled timing, i.e., waiting for the participant to do the next action before triggering the next response. All participants received the same stimuli (according to which experimental group they were in) apart from the issue of timing. E.g., the character’s response to shock 5 would have been the same for all participants in the “visible condition”.


QUERY:
4) Excluded participant
In the "Recruitment" section, it's reported that one participant "was eliminated who had stopped after only 5 shocks." This was a participant in the Visible Condition.

In their (29 Dec 2006) response to Eveaustria's questions, the authors state:
"Another participant withdrew after only 6 shocks, and we were suspicious about that."

Was it 5 shocks or 6 shocks that the participant gave before withdrawing, and which condition ?
This strikes me as an important distinction because it was in response to the 11th question (which should elicit the 6th shock) that the learner first protested, shouting “Stop the experiment!” and responding to the experimenter's statement with “Don't listen to him, I don't want to continue!”
This certainly seems a likely place for participants to first consider withdrawing from the experiment.
At any rate, whether this excluded participant withdrew before or after administering the 6th shock after the learner's outburst, why was this considered grounds for exclusion from data analysis?

RESPONSE:
The one in the paper is correct, the other was written in error. It makes no difference to the results whether this participant is included or not. I decided to exclude on safety grounds since the behaviour of this participant was extreme in relation to all the others.

QUERY:
Final comments:
Perhaps this paper simply follows a format I am unfamiliar with, but I thought the methods and procedures could have been better organized. I was confused by the fact that some information was included in the subsection "Procedures" under "RESULTS" and other information was not found until another subsection labeled "Procedures" under "MATERIALS AND METHODS." More consolidation would have been helpful.

RESPONSE:
Instructions given in PLoS ONE guidelines were followed, according to our interpretation of them.

Thanks for your helpful comments and questions.