Bridging the Mechanical and the Human Mind: Spontaneous Mimicry of a Physically Present Android

Galit Hofree; Paul Ruvolo; Marian Stewart Bartlett; Piotr Winkielman

doi:10.1371/journal.pone.0099934

Abstract

The spontaneous mimicry of others' emotional facial expressions constitutes a rudimentary form of empathy and facilitates social understanding. Here, we show that human participants spontaneously match facial expressions of an android physically present in the room with them. This mimicry occurs even though these participants find the android unsettling and are fully aware that it lacks intentionality. Interestingly, a video of that same android elicits weaker mimicry reactions, occurring only in participants who find the android “humanlike.” These findings suggest that spontaneous mimicry depends on the salience of humanlike features highlighted by face-to-face contact, emphasizing the role of presence in human-robot interaction. Further, the findings suggest that mimicry of androids can dissociate from knowledge of artificiality and experienced emotional unease. These findings have implications for theoretical debates about the mechanisms of imitation. They also inform creation of future robots that effectively build rapport and engagement with their human users.

Citation: Hofree G, Ruvolo P, Bartlett MS, Winkielman P (2014) Bridging the Mechanical and the Human Mind: Spontaneous Mimicry of a Physically Present Android. PLoS ONE 9(7): e99934. https://doi.org/10.1371/journal.pone.0099934

Editor: Stefano Federici, University of Perugia, Italy

Received: April 19, 2013; Accepted: May 20, 2014; Published: July 18, 2014

Copyright: © 2014 Hofree et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: This work was supported in part by NSF grants IIS-0808767 and SMA-1041755. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

A key task for psychology and related disciplines is to identify mechanisms that allow humans to understand others, quickly respond to them, and coordinate mutual actions. One mechanism supporting these capacities is imitation [1], [2]. The core form of imitation is spontaneous mimicry, where merely observing another individual's behavior elicits a corresponding behavior in the observer without being instructed to do so [3]. The most basic form of such mimicry is the mirroring of emotional facial expressions—a phylogenetically ancient form of intra-species communication that occurs in humans and in some other primates [4]. As shown by a large body of research, the mere viewing of expressions can spontaneously and rapidly activate congruent facial muscles such that perceivers, for example, smile in response to a smile and frown in response to a frown [5], [6]. Such behavior can, under some conditions, support recognition of emotional states (e.g., [7], [8]). In general, spontaneous mimicry promotes empathy, rapport, contagion, and social coordination [9], [10] via a variety of psychological and neural mechanisms [9], [11], [12].

Researchers interested in imitation have long utilized robots as research tools. Such research has shown, for example, that voluntarily mimicking robots leads to more positive interactions [13], [14] and that the speed of rudimentary human motor responses is influenced by compatibility with a movement of a simple robotic hand [18]. Critically, new discoveries about the underlying mechanisms, as well as social consequences of imitation and mimicry, are now made possible by developments of so called “hyper-realistic androids”—robots that possess a humanlike face and produce believable emotional expressions. As elaborated shortly, evidence of spontaneous human mimicry of such androids would have important implications for major theories of mimicry. Moreover, such evidence would be important for informing theoretical questions of robotics, such as synchronization between robotic and human agents, the role of emotions in human-robot interaction, and the role of androids' physical presence in cognitive and emotional responses. Answers to these questions are also practically important since such agents are currently being developed, at great cost, for households, education, customer service, and care for the elderly and disabled. Further, the lifelike appearance of these robots is presumed to enhance human welfare [15] in situations where the natural social interaction between robots and humans is a crucial component to success in these efforts [16]. Interestingly, for practical purposes androids often resemble well-known humans (e.g., Repliee, various actroids, and Hanson's Einstein). Evidence of spontaneous mimicry of such realistic androids could facilitate creation of agents that effectively build social rapport and create a new level of engagement with their human users, using a naturalistic non-verbal, high-bandwidth robot-human communication channel.

Theoretical Accounts that Predict Mimicry of Androids

As mentioned, our main focus here is on how hyper-realistic androids allow insights into the mechanisms of spontaneous mimicry. Note that different theoretical accounts make different predictions regarding the emergence and the exact nature of android-human mimicry. Later we will discuss accounts that doubt the very possibility of android mimicry. But, first, let us consider some theoretical accounts that predict robust human mimicry of such agents. Note that some of these accounts are generally concerned with motor mimicry, whereas others focus more on emotional processes.

One general ‘motor’ account, termed associative sequence-learning (ASL), assumes that mimicry reflects relatively low-level, automatic operations involved in sensorimotor mapping [17]. According to ASL, facial mimicry represents a subclass of general stimulus-response compatibility effects, in which observation of a task-irrelevant action facilitates a similar action in the perceiver, by virtue of repeated previous pairings of perception and motor execution. In short, as long as the novel stimulus is reasonably similar to stimuli that were previously associated with specific motor actions, it should trigger these same actions. The face of a hyper-realistic humanoid robot is certainly such a stimulus, and indeed the prediction of robust mimicry is consistent with early work reporting imitation of robotic hands [18].

Another account that predicts robust mimicry of an android's expression assumes that mimicry reflects involvement of a sensorimotor loop aimed at facilitating stimulus recognition. Specifically, facial mimicry represents an automatic embodied simulation that provides motor feedback to the perception process, and thereby helps distinguish subtle differences in the observed emotional expression [19]. According to this account, as long as people are trying to recognize a face, regardless of whether this faces is actually human or robotic, they should use such an embodied simulation strategy.

The above accounts (ASL and automatic embodiment) focus on general processes, and assume that the process underlying facial mimicry is essentially motoric in nature. Other accounts that predict robust mimicry of an android's expressions focus on emotional processes. One such account, sometimes termed the “affect-matching account”, assumes that seeing facial expressions automatically triggers corresponding affective responses in the perceiver [20]. For example, seeing a smile induces positive affect, which then triggers a smiling expression. If an android produces an expression that is highly similar to an actual human smile, it may elicit positive affect in the human perceiver, and thus trigger smiling. It is worth noting that while the motor and affect accounts agree that an android can trigger facial mimicry, the two theories make slightly different predictions regarding the exact nature of corresponding muscle responses. The motor accounts assume that the process is strictly imitative and thus the produced expression should closely mirror the shape and dynamics of the observed expression. The affect-matching account suggests that the shape and dynamic of a facial response reflects the internal nature of the triggered emotional process. In this sense, “mimicry” is a slightly misleading term, though often used, for such “matched” responses, as the similarity between perception and production comes from a match in affective state rather than motoric features. Accordingly, the affect-matching account predicts that facial “mimicry” responses can sometime disassociate from the perceived expression, as when observation of a fearful expression triggers negative affect expressed by facial frowning – a feature absent in the eliciting “fearful” stimulus [21]. We will return to this issue in the discussion, and only note that we use the term mimicry technically, without committing to a particular interpretation. For now, the important point is that all accounts discussed so far predict that humans will mimic expressions of an android, especially a state-of-the-art android that closely resembles a human being.

Perception of Human-Likeness in Androids

But what does it mean for an android to “resemble a human being?” Most views agree that some perception of an android's “human-likeness” plays a necessary role. However, they disagree on what aspects of being “humanlike” are essential. Some views, which we will explicate later, emphasize the role of psychological similarity, where the critical variable is the perception of the ability to have “intentional states, such as beliefs, desires, and emotions” [22]. Other views emphasize the role of physical similarity, where the critical variable is the perception of overlap in external appearance between humans and androids. This sense is closest to the Latin meaning of anthropomorphism, which is about human form or shape [18], and might also be informed by perception of human motion [23]. The possible importance of anthropomorphic form or motion is consistent with recent findings from the human-robot interaction (HRI) field demonstrating that people respond facially to non-human agents that are physically similar to humans. For example, people react with similar facial expressions to a video of a robotic face when asked to identify the emotion it is displaying [24]. Another study reports that people respond to a head gesture mimicking chimpanzee robot with similar gestures [14]. Avatars—virtual human characters, can also elicit facial mimicry in observers [25]–[27]. In short, if anthropomorphism is the key to unlocking mimicry and relatively superficial appearance-based humanlike features can trigger automatic mimicry, then an ultra-realistic android should be even more likely to induce mimicry reactions.

Critically, the perception of human-likeness may involve more than perception of psychological and/or physical similarity. Recent work emphasizes the role of the actual physical “presence” of the robot. In fact, some research suggests that merely sharing the same physical space with an android influences how humanlike it is perceived to be [28]. This is important because it suggests that actual presence may enhance the emergence of mimicry. More generally, the role of presence highlights the theoretical and practical value of investigating the impact of actual, physically-present robots. While avatars on a computer screen can be quite realistic, they hardly approach the details of an android. After all, these androids often have actual hair, human-like skin, clear eyes, and other salient features that are normally encountered only in real, living humans. These androids are also collocated in the human's personal space — they are tangible, within human reach, and have the human within their reach. In addition, while people are very familiar with robots and animated on-screen avatars (since TV and movies provide an alternative reality in which robots and avatars are plausible), they are unfamiliar with physically present androids. The novelty of such an experience, as well as the more concrete representations of the android brought about through spatial proximity [29], might further highlight these humanlike features. Nonetheless, the experience of presence appears to be a complex phenomenon, and it is yet unclear how visual realism contributes to it [30]. Interactions with real robots might therefore lead to qualitatively different reactions. So, while the earlier work on nonhuman agents leaves open whether subjects will imitate a realistic android, it is certainly suggestive of this possibility.

Theoretical Accounts Predicting No Mimicry of Androids

As mentioned earlier, there are some alternative theoretical perspectives that speak against the possibility of human mimicry of androids. One prominent view emphasizes that mimicry is a tool for understanding the observed agent's emotions and intentions from “within,” via perceiver's simulation of the agent's mental states (for a review, see [31]). Under a strong interpretation of this view, mimicry will occur only if the perceiver believes that the observed agent actually possesses mental states (otherwise, there is nothing to find out from “within”). In fact, there is some evidence that “psychological anthropomorphism,” (or the belief that that an agent shares human-like intentions and emotion) predicts many emotional responses to other agents [32], [33]. According to that view, people should not mimic an android, if they indeed believe that it lacks intentionality and emotion.

Another view that predicts an absence of android mimicry emphasizes the role of emotional unease humans might experience when confronted with an android – an eerily human-like, yet clearly artificial being [34]–[36]. This prediction is based on evidence that a human's negative attitudes towards a model generally reduce or even reverse mimicry [19], [25], [37], [38]. Note that emotional unease might be exacerbated when interacting with an android face-to-face, due to a salient mismatch between its humanlike appearance and robotic features (such as mechanical sounds or jerky un-humanlike movements). In short, negative emotions should lead to disengagement and discourage relating to the android in the form of mimicry.

Current Research

Our research examined the issues discussed above in two studies that tested mimicry of a hyper realistic android. Specifically, we explored whether people will spontaneously mimic such an android, and whether the strength of mimicry depends on the android's actual physical vs. virtual presence (via video). In addition, we tested how mimicry depends on (i) the perceptions of human-likeness and intentionality, and (ii) the feelings of emotional unease about the android. In both studies, we employed Hanson's Einstein, a state-of-the-art android programmed to perform realistic human expressions (see sample video of Einstein: http://pages.ucsd.edu/~pwinkielman/einstein_happy_angry.mov). In Study 1, we compared participants' mimicry reactions in response to videos of the android and to a matched human control making the same expressions. In Study 2, we examined participants' reaction to a physically present android—where humans sit face-to-face with the actual robotic agent. In addition, we had participants assess the android on human-likeness, emotional comfort, as well as intentionality.

Our predictions were as follows. Based on theoretical views emphasizing the automaticity of mimicry-related processes, we predicted that the android would elicit spontaneous human mimicry. Furthermore, such mimicry should occur even when participants do not perceive the android as having intentionality, and even when participants experience psychological discomfort with the agent. Based on the idea that mimicry requires a sense of similarity and relatedness between the self and the other [16], [18], we also predicted that mimicry should be magnified by perceptions of human-likeness as well as the physical presence of the robot.

Study 1

Methods

Robot design.

Both studies used an android manufactured by Hanson Robotics. This is one of the most advanced androids available to study human-robot interactions, and its facial behavior has been programmed according to the highest standards in the field [39]. Its face, shown in Figure 1A, is made of skin-like materials and is actuated by 31 servos (motors) programmed to perform facial movements that closely match human expressions, such as happiness and anger (Figure 1B/C, see also File S1). As with other hyper-realistic androids, its appearance is personalized by resemblance to a well-known human — in this case, Albert Einstein.

Download:

Figure 1. Linking android and human facial expressions – servos vs. muscles.

(A) Einstein the robot, as the participants saw him. Robot expressions of (B) happy and (C) angry. (D) Relationships were investigated between the actuators shown in blue, and facial muscle activity in the corrugator and zygomaticus, as indicated by the red arrows. The subject in the photograph has given written informed consent, as outlined in the PLOS consent form, to publication of their photograph.

https://doi.org/10.1371/journal.pone.0099934.g001

Videos.

We created 6 second videos of the robot and an aged-matched human control making happy and angry expressions. To make the videos fully comparable between agents and emotions, both in terms of the temporal dynamics and the intensity of the expression, we made two versions of each expression: (i) natural and (ii) intensity-matched. The natural videos were created by asking the control human to watch the android's expressions and mimic them as closely as possible. This matches the videos on subjective perceptions of similarity. The intensity-matched videos were edited in such a way that the peak intensity of the expression matched between the android and the human control (as judged by a FACS expert). The onset, peak time, and offset of the expressions were the same, thus formally standardizing our videos both across time and intensity.

Pretest - Anthropomorphic Ratings of Android and Control Videos.

We first tested whether people attribute mental states to the android, which according to some views discussed earlier, is essential for mimicry. We ran this pretest separately so as not to interfere with the measurement of spontaneous mimicry reactions. In this pretest, we used the widely used Individual Differences in Anthropomorphism Questionnaire, or IDAQ [40]. It includes questions regarding the mental states (e.g., intention, free will, and emotion) of technological devices, as well as non-human animals, and natural entities. In addition, we constructed a modified IDAQ with the same questions regarding the android and the human control. During the pretest study, 392 separate UCSD participants (300 female) first filled out the standard IDAQ questionnaire. Next, participants watched happy and angry (matched-intensity) videos of both the android and the control (counterbalanced) and then filled out a modified IDAQ about the android and the human control.

The results of this pretest show that participants rated the human control as overall higher in intentionality than the android (M_android = 3.55, M_control = 7.39, t(382) = 20.17, p<0.0001, d = 2.06). In fact, participants perceived the android as significantly less capable than the human control of mental states such as free will, consciousness, and emotion (all ps<0.0001), rating him near the low end of the scale for each of these states. For comparison purposes, the android was rated as possessing more free will than a car, but less than a fish (M_android = 3.19, M_car = 0.72, M_fish = 5.7). Android consciousness was rated above that of an average robot, but less than a reptile (M_android = 3.27, M_robot = 1.45, M_reptile = 6.43). The android was rated as experiencing more emotions than a television set, but less than a cheetah (M_android = 4.25, M_TV = 1.21, M_cheetah = 7.14). The human control was rated above all of the examples used in the IDAQ, as expected. Interestingly, individuals' responses to IDAQ questions regarding technological devices predicted the difference between the human and android ratings. Specifically, those who considered technology as being more anthropomorphic also rated the android and the human control more similarly on anthropomorphism (b = −0.59, t(380) = −5.39, p<0.0001, d = 0.55). Gender of the participants had no effect on these ratings. It appears that individuals' general tendency to anthropomorphize technology was associated with attributing more internal humanlike properties to the android (e.g., intentions and emotions). Overall, the critical point here is that participants rate the android low in intentionality.

Participants in the main experiment.

Forty-eight undergraduates from University of California, San Diego (UCSD) participated in this experiment (30 female and 18 male). The research protocol was approved by UCSD Institutional Review Board and all participants provided written informed consent. Twelve subjects were excluded from EMG analyses due to corrupt data, 2 were excluded from the experiment due to their familiarity with the android or human control, and another 5 were excluded from the experiment because they guessed that we were measuring facial expressions or mimicry specifically.

Ratings during the experiment.

We were especially interested in how mimicry depends on specific kinds of perception: (i) the sense of “humanlike” quality of the android and (ii) the sense of comfort with the android. Thus, we had participants make such ratings, for both the android and human (intensity-matched) videos, before and after the mimicry phase. In addition, after the experiment, participants also rated their arousal, negative feelings, and positive feelings. The android was also specifically rated on “creepiness” (specifically, we asked how creepy/scary/weird subjects found the robot). Answers to these questions were measured on 9-point Likert scales, ranging from 1 (not at all), through 5 (moderately), to 9 (very).

Procedure.

As just mentioned, participants first rated both agents (robot and control) on comfort and humanlike attributes (presentation and ratings were counterbalanced). Then, they proceeded to the mimicry part of the study, which was modeled after a classic paradigm [5], [6]. Participants viewed randomly interspersed videos of the android and the control, each displaying happy and angry expressions. As in previous research, there were two phases. In the first, spontaneous phase, participants were instructed to simply observe the videos, without receiving any instructions or encouragement to mimic. In the second, intentional phase, participants were told to “try and make the same face as the object in the video.” This intentional condition was included to ensure that the human and androids' expressions are easily visible and principally imitable and that our EMG paradigm can accurately detect mimicry. The order of these two conditions was kept constant so as to avoid the introduction of an implicit expectation for mimicry behavior in the spontaneous phase.

In each condition, there were separate blocks of natural videos, along with matched-intensity expression videos, each with 40 trials. Following the mimicry tasks, participants again rated both agents on various attributes (see above). We gauged participants' mimicry behavior using facial electromyography (EMG), a technique that measures electrical changes in underlying muscle activity, thus allowing for fast and sensitive online assessment of participants' facial reactions to the android's expressions. Our methods follow the official and published standards for EMG recording, collection, analyses and data presentation [41], [42]. Following these standards, electrodes were placed over the cheek muscle (zygomaticus major) and the brow muscle (corrugator supercilii) (for more details, see File S1).

Results

Pre-experiment ratings.

As expected, participants found the android to be significantly less humanlike (M_android = 4.32, M_control = 7.42, t(37) = 8.89, p<0.0001, d = −1.44), and felt less comfortable viewing his videos than those of the control (marginally significant two-tailed t-test: M_android = 4.11, M_control = 4.68, t(37) = 1.83, p = 0.08, d = 0.29). There were no gender differences in these ratings.

Post-experiment ratings.

Participants' responses were consistent with their pre-experiment ratings of the android and the control on comfort and humanlike attributes. The only exception was ratings of how humanlike they found the control; they rated the control as significantly more humanlike following the experiment (M_pre-mimicry = 7.42, M_post-mimicry = 8.05, t(37) = 2.27, p = 0.03, d = 0.38). In addition, they rated the android as significantly less arousing (M_android = 1.95, M_control = 2.58, t(37) = −2.77, p = 0.009, d = −0.47) and less positive (M_android = 2.34, M_control = 3.16, t(37) = −2.75, p = 0.009, d = −0.45), but not more negative (minimal gender differences were found, see File S1). Finally, they also rated the android as more than moderately creepy (t-test comparing mean to 5, which was our scale “moderate” midpoint): M_creepy = 5.48, t(37) = 2.11, p = 0.04, d = 0.34). In summary, our participants find the android in the videos not very humanlike, especially when compared to the human control, and feel less comfortable viewing it, to the point of finding it rather “creepy”.

EMG data.

Analyses focused on the following predictions. If humans mimic, then shortly after the appearance of the target's expression, they should show greater EMG activity in the brow area for anger and cheek area for happiness. Critically, such mimicry should occur in the spontaneous condition. As discussed earlier, spontaneous mimicry of the android should be modulated by individual ratings of human-likeness [17], [18]. The intentional condition should obviously produce strong mimicry, in response to both the android's and the human's expressions.

Specifically, we compared EMG activity between happy and angry trials, for both agents and both muscles. We expected mimicry-related EMG activity to be as follows: activity of the corrugator muscle should be elevated during angry trials, as compared to happy trials, while the reverse should occur for the zygomaticus muscle. Note that if mimicry closely matches stimulus characteristics, there should also be some zygomaticus activity during angry trials. This is because the android's angry expressions also contain a “grimace” involving its cheek (see sample video linked in the Introduction).

Data were analyzed by comparing happy and angry trials separately for each muscle using a repeated-measures MANOVA over 500 millisecond intervals of a 6 second trial (i.e., 12 time points). Because spontaneous mimicry reactions can occur rapidly [5], [20], we also conducted similar MANOVAs over 200 millisecond intervals of the first second of the trial (i.e., 5 timepoints). Time was included as a factor in all analyses to account for changes in EMG responses over the course of the trial. Gender effects were tested and are reported separately in File S1. Before the main analyses, we collapsed responses to the matched-intensity and natural videos, as preliminary analyses did not reveal any effect of these conditions. Next, we first present our analyses conducted on each agent (android, human) separately, and then discuss an omnibus ANOVA that includes agent type as a factor. In addition, we tested the role of humanlike perception of the android. To do so, we conducted mimicry analyses on participants who responded high or low on the human-like rating, using a median split. Finally, to test whether emotional unease decreases mimicry, we conducted similar analyses on participants who responded high or low on comfort ratings [25].

Spontaneous mimicry.

We first tested whether we could replicate the standard phenomenon of spontaneous mimicry in the human control condition. Figure 2 shows that participants clearly mimicked the human control, in both muscles. As shown in Figure 2 top left, zygomaticus activity during happy trials significantly increased from the beginning of the trial to peak at 1.5s, t(28) = 2.22, p = 0.03, d = 0.84. During angry trials, zygomaticus activity did not differ from baseline, as demonstrated by a non-significant contrast comparing activity across all 12 timepoints to zero, t(11) = 1.32, p = ns. Correspondingly, the zygomaticus demonstrates a significant Emotion × Time interaction, due to participants smiling more and earlier at happy expressions, than angry expressions, F(11,308) = 2.67, p = 0.003, partial η² = 0.09. Analyses of rapid response (200 ms intervals) demonstrate that this mimicry reaction begins within the first second (Emotion × Time interaction: F(4,112) = 3.37, p = 0.012, partial η² = 0.08).

Download:

Figure 2. Spontaneous mimicry in both muscles for the human control and android videos, for those who rated the android high on humanlikeness.

Zygomaticus activity (top) and corrugator activity (bottom) across 6 second trials in response to a human control (left), and in response to the android (right). Those who rated the android high on humanlikeness (right) showed mimicry reactions in the zygomaticus muscle (top right).

https://doi.org/10.1371/journal.pone.0099934.g002

The analysis of the corrugator muscle for the Human Control condition revealed a significant main effect of Emotion, such that participants frown more to angry than happy expressions, F(1,28) = 4.06, p = 0.05, partial η² = 0.13. As illustrated in Figure 2 bottom left, there is also a significant Emotion × Time interaction, where the corrugator is activated more in the beginning portion of the trial, peaking at 1s and diminishing by 3 s, F(11,308) = 2.61, p = 0.003, partial η² = 0.09. A MANOVA conducted over the 200 ms intervals of the first second of the trial reveals both a main effect of emotion (F(1,28) = 5.75, p = 0.023, partial η² = 0.17) and a significant Emotion × Time interaction (F(4,112) = 4.71, p = 0.002, partial η² = 0.14), demonstrating that corrugator associated mimicry also occurred very early into the trial. In summary, we observed robust spontaneous mimicry to human facial expressions.

Next, we analyzed mimicry to the android. As mentioned earlier, we were especially interested in how perception of the android's similarity to humans might influence mimicry. For this reason, we also tested if mimicry depends on whether participants initially rated the android as low or high in human-likeness (median-split). Indeed, these two groups react to the android very differently. This is most evident in the activity of the zygomaticus muscle. A MANOVA including low/high human-likeness as a factor reveals a significant Human-Likeness × Emotion interaction, F(1,27) = 10.94, p = 0.003, partial η² = 0.29. This difference in reaction is already apparent within the first second of the trial (Human-Likeness × Emotion: F(1,27) = 9.3, p = 0.005, partial η² = 0.29). When examining just those participants who rated the robot highly on human-likeness, we found that they show significant signs of mimicry (main effect of Emotion: F(1,17) = 6.91, p = 0.02, partial η² = 0.29), smiling more to happy expressions than to angry expressions (Figure 3 top right). However, these reactions to the android are still weak compared to the human stimulus. This is formally demonstrated by an analysis that added Agent type to the above MANOVA conducted only on participants who rated the android high in human-likeness and revealed an Agent × Emotion × Time interaction, F(11,187) = 2.457, p = 0.007, partial η² = 0.13. A MANOVA with human-likeness included for corrugator activity did not yield any significant effect of Emotion. Interestingly, corrugator responses for both expressions went up from the baseline (contrast comparing activity across all 12 timepoints to zero: happy: t(11) = 16.69, p<0.0001, d = 10.06; angry: t(11) = 16.79, p<0.0001, d = 10.12). Similar analyses including a median split of comfort ratings revealed no significant interaction with mimicry reactions in either muscle.

Download:

Figure 3. Intentional mimicry in both muscles for the android and human control.

Zygomaticus activity (top) and corrugator activity (bottom) across 6 second trials in response to the human control's facial reactions (left), and in response to the android (right).

https://doi.org/10.1371/journal.pone.0099934.g003

Finally, we conducted a full repeated-measures MANOVA comparing happy and angry trials that also included Agent (android and human control) as a factor. This analysis revealed some overall differences between reactions to the robot and the human. The corrugator muscle shows a significant main effect of Agent, such that participants frown more overall to the human, F(1,28) = 9.37, p = 0.005, partial η² = 0.25. Most likely this reflects that participants only mimicked happy expressions of the android (and not angry expressions), while they mimicked both the angry and happy expressions of the human. There was also a significant Agent × Time interaction for the corrugator (F(11,308) = 2.5, p = 0.005, partial η² = 0.08) and a significant Agent × Emotion × Time interaction for the zygomaticus, F(11,308) = 2.35, p = 0.009, partial η² = 0.08. No effects of gender were found for the spontaneous condition for both muscles. Conducting similar MANOVAs over the first second of the trial demonstrated that differentiation in response to the agents begins very early. Similar to the full trial MANOVAs, a main effect of Agent was found in the corrugator (F(1,28) = 4.20, p = 0.05, partial η² = 0.13), where in the zygomaticus a significant Agent × Emotion interaction (F(1,28) = 10.94, p = 0.03, partial η² = 0.16) demonstrates that participants begin smiling early to the control, but not to the android. Overall, these results suggest that participants are quicker to react to the expressions of the human control, as compared to the android.

Intentional mimicry.

Figure 3 shows intentional EMG activity across all time-points in a trial in both muscles (for both the android and the control). It is clear from this figure that both agents elicited robust mimicry. This was confirmed by statistical analyses of EMG activity for the android and the control separately.

Figure 3 top right illustrates the results for the android. On the zygomaticus muscle, there was strong, significant main effect of Emotion, F(1,28) = 34.36, p<0.001, partial η² = 0.55, as well as a significant Emotion × Time interaction, F(11,18) = 6.49, p<0.001, partial η² = 0.80. Participants are clearly smiling more to android's happy expressions than angry expressions. Interestingly, there is also some evidence that in this condition participants are also grimacing in response to angry expressions (linear contrast comparing to a zero baseline: t(11) = 8.04, p<0.0001, d = 2.41). Also, it appears that zygomaticus activity occurs earlier and stronger for happy expressions than for angry expressions. As illustrated by Figure 3 bottom right, the corrugator muscle also shows a significant main effect of Emotion, (F(1,28) = 126.42, p<0.001, partial η² = 0.82), along with an Emotion × Time interaction, F(11,18) = 16.83, p<0.001, partial η² = 0.91. As expected, participants frowned to the android's angry expressions but not its happy ones.

For the human control, we saw similar results, as shown in Figure 3 left top/bottom. That is, we observed a main effect of Emotion for both the corrugator and zygomaticus muscle (corrugator: F(1,28) = 170.67, p<0.001, partial η² = 0.86; zygomaticus: F(1,28) = 43.60, p<0.001, partial η² = 0.61), as well as significant Emotion × Time interaction for both muscles (corrugator: F(1,28) = 14.05, p<0.001, partial η² = 0.90; zygomaticus: F(11,18) = 10.52, p<0.001, partial η² = 0.87). These intentional mimicry reactions are initiated early in the trial, as can be seen in analyses conducted over the first second of the trial. We find both a main effect of Emotion and Emotion × Time interaction in both the zygomaticus and corrugator muscles, for both the android and the control (see File S1).

The preceding analyses make clear that intentional mimicry was robust to both the human and the android. Nonetheless, when including Agent into the MANOVA (as before), we again found evidence for a difference in reaction to the android and the human control. The zygomaticus showed a significant main effect of Agent (F(1,28) = 57.63, p<0.001, partial η² = 0.67), a significant Agent × Time interaction (F(11,18) = 9.21, p = 0.005, partial η² = 0.71), and a significant Agent × Emotion × Time interaction (F(11,18) = 2.7, p = 0.03, partial η² = 0.63), as well as the expected effect of Emotion (F(1,28) = 131.62, p<0.0001, partial η² = 0.67) and Emotion × Time interaction (F(11,308) = 41.35, p<0.0001, partial η² = 0.60). In short, while participants mimic both the android and the control, they are clearly reacting more to the expressions of the human control than to that of the android. More specifically, they smile more to the control's happy expressions and grimace more to the control's angry expressions. In terms of the corrugator, it showed the expected effects of Emotion (F(1,28) = 175.59, p<0.0001, partial η² = 0.86) and Emotion × time interaction, F(11,308) = 71.32, p<0.0001, partial η² = 0.72. More interestingly, the corrugator also demonstrated a significant Agent × Time interaction (F(11,18) = 5.62, p = 0.001, partial η² = 0.25) and a significant Agent × Emotion × Time interaction (F(11.308) = 2.98, p = 0.001, partial η² = 0.10). Figure 3 (bottom panels) illustrates that participants' frowning response to the human control's anger expression peaks for a longer duration than their frowning response to the android. Note also that participants show differentiated reactions to the two agents early in the trial in both muscles, as can be seen in analyses conducted over the first second of the trial (see File S1). It is also worth noting that some gender effects were found for both muscles in this condition (see File S1).

Discussion

This study, in which participants watched videotaped facial expressions of an android and a control human, revealed several interesting findings. The analyses of ratings revealed that participants clearly find the android less positive than the human control, and even see him as more than moderately “creepy.” Moreover, they find the android less humanlike, and attribute less intentionality to him, compared to the human control. It is also worth highlighting again that in the pretest ratings, the android was rated as less conscious than a reptile, having less free will than a fish, and less emotion than a cheetah. In short, in terms of intentional states, our android was rated low.

Still, some participants did spontaneously mimic the android. Interestingly, those participants also rated the android high on human-likeness. The ratings of emotional discomfort, or intentionality, did not modify spontaneous android mimicry. As such, the results of Study 1 reveal that some aspect of “human-android” relationship plays a critical role in spontaneous mimicry. Further, these results suggest that mimicry can dissociate from knowledge of artificiality and experienced emotional unease. Nevertheless, the android mimicry was weaker than spontaneous mimicry of a human, and was evident only in the zygomaticus muscle. Finally, the intentional mimicry of the android was very robust, albeit also weaker than of the human. All of this suggests that the android could, in principle, elicit robust spontaneous mimicry, given the right conditions, as explored next. We will return to the larger theoretical implications of these results for different mimicry theories in the general discussion.

Study 2

In study 1 we found that people, under some limited conditions, will spontaneously mimic a hyper-realistic android. We proposed that the critical variables relate to individuals' general sense of relation to the android, and not to their emotional comfort, or their belief in its intentionality. If so, one would expect more robust android mimicry in a situation where participants personally encounter the robot. Specifically, as discussed earlier, previous research suggests that direct experience – the robot's actual physical presence – may enhance mimicry. One reason is that direct presence enhances the salience of the android's realistic visual appearance, possibly increasing the sense of human-likeness. In other words, when actually confronted with the android, the salience of humanlike visual features may dominate over any perception of artificial robotic movement. In contrast, when watching that same robot on a screen, the artificiality in movement may be more pronounced (humanlike features appearing less salient on video), leading to weaker mimicry reactions. Interestingly, the same change in salience of detectable human features might also lead to more extreme negative reactions, amplifying the conflict between human-likeness and artificiality. Einstein, the android used in these studies, is actually a disembodied head and shoulders, placed on a platform (see Figure 1A). Although his features are very humanlike, the fact that he is a disembodied figure might cause more discomfort. Thus, testing reactions to the android in face-to-face setting can address questions regarding the relative impact of human-likeness and emotional unease on spontaneous mimicry. In addition, using a real “live” robot opens opportunities to measure mimicry in more methodologically sophisticated ways, such as synchrony between robot facial servos and human facial muscles. For all these reasons, we ran two pretests assessing people's impressions of a physically present android, and, in the separate main experiment, tested people's facial mimicry of a physically present android.

Methods

Pretest 1 – free descriptions of the android.

This pretest measured people's spontaneous impressions of the android in a face-to-face interaction. We told 36 separate participants (11 female) that they would be interacting with an android named Einstein, after which they would answer some questions about him. They were placed in a chair facing the robot and asked to freely describe “the android in front of you.” Frequency analysis of these descriptions revealed that participants found the android highly realistic and humanlike. Specifically, 51% used exact words or close synonyms of “realistic”, “real”, “human-like”, “life-like”, and “person.” Spontaneous descriptions for 51% of the participants also included an emotional reaction. As expected, most of these reactions (78%) imply unease about the robot and include words (or close synonyms) of “scary”, “creepy”, or “weird.”

Pretest 2 – ratings of the android.

This pretest assessed people's general beliefs about intentionality and technology, and their perception of the android's mental states after a face-to-face interaction. Participants were a separate group of 203 undergraduates (148 female). The participants first, and in a separate room, completed the general IDAQ questionnaire [40]. Next, they were sat in a chair facing the android where they examined him making two facial expressions: a happy expression and an angry expression. Participants were then asked to answer the same questions used in Study 1 (i.e., how humanlike is the android, how comfortable do they feel, how aroused/excited do they feel, how sad/bad do they feel, how happy/good do they feel, and how creepy/scary is the android), along with additional questions regarding his mental states. Gender differences in these ratings are reported in File S1.

Analyses on the android-related items showed that it was rated as above moderately humanlike (t-test comparing mean to the midpoint, 5, signifying moderately humanlike: M = 5.74, t(159) = 4.49, p<0.0001, d = 0.71), but also creepy (M = 6.76, t(159) = 10.65, p<0.0001, d = 1.69). Note, however, that intentionality scores for the android were particularly low (M = 1.05, s = 1.55), suggesting that although it appears human-like, participants do not endow the android with human mental states. Individual differences in intentionality ratings of the android were predicted by individual differences in general intentionality scores (b = 0.55, t(157) = 7.3, p<0.0001) and specific technology-related intentionality questions (b = 0.44, t(157) = 9.45, p<0.0001). As in the previous study, these results suggest that an individual difference in anthropomorphisation of technology in general may lead to anthropomorphisation of the specific android.

In addition, we compared ratings of the directly present android to those of his videos from Study 1. This allows us to examine whether the physically present android appears more humanlike than the videos of the same android (Figure 4). Interestingly, the “live” android was found to be even less anthropomorphic (using the aforementioned IDAQ questionnaire [40]) than his video counterpart (M_present = 1.045, M_video = 3.56, t(536.15) = 12.17, p<0.0001, d = 1.05), and participants were less comfortable viewing him (M_present = 3.28, M_video = 4.11, t(53.61) = 2.31, p = 0.02, d = 0.63). However, they found the “live” android significantly more humanlike (M_present = 5.74, M_video = 4.32, t(56.14) = −3.81, p = 0.0004, d = 1.02), although still less humanlike than the control (M_present = 5.74, M_control = 7.42, t(61.94) = 4.97, p<0.0001, d = 1.26). This suggests that interacting face-to-face with the android leads to heightened impressions of human-likeness, albeit with decreased attributions of internal human-like states – an interesting juxtaposition to which we come back in the general discussion.

Download:

Figure 4. Comparison of ratings for the present android as a robot, on video, and human control.

Participants rate the present android as significantly lower in intentionality than his video counterpart, and the human control. Participants feel the least comfortable with the physically present android, yet rate him more humanlike than his likeness in the video. Note: ratings for comfort and humanlike were made on a 1-9 scale, while those on intentionality were made on a 0-10 scale in accordance with the IDAQ [40].

https://doi.org/10.1371/journal.pone.0099934.g004

Participants in the main mimicry experiment.

Participants were 23 University of California, San Diego undergraduates (9 female and 14 male). The research protocol was reviewed and approved by the University of California, San Diego Institutional Review Board and all subjects provided written informed consent. Four subjects were excluded due to corrupt data.

Robot servo measurement.

As mentioned in Study 1, the android was programmed to make human facial expressions using 31 different servos. Of these, several were responsible for making happy and angry faces, which were the expressions used in these mimicry experiments. Figure 1D highlights the relevant servos and the tested links between android and human facial movement. To enable quantitative measurements of synchrony between the android's and a human's facial expressions, we calculated the electrical input to the critical ‘brow’ servo and the critical ‘cheek’ servo (see Figure 1A/D and File S1).