Introduction

It has been shown numerous times that when congruent visual and auditory cues are processed together perceptual accuracy is enhanced in both normally-hearing (NH) and in hearing-impaired individuals (e.g. [1]–[13]). In contrast, several investigations have demonstrated that when incongruent visual and auditory cues are processed together, audiovisual interactions seem to occur differently in hearing-impaired individuals, compared to NH. Specifically, audiovisual perception is dominated by auditory information in the NH, whereas it is dominated by vision in hearing-impaired individuals that are using a cochlear implant (CI). For example, a number of studies that used a McGurk effect paradigm [14] have shown that CI users are able to integrate auditory and visual information adequately [12], [15], [16] but that they essentially refer to the visual cues when incongruency makes integration difficult [11], [17], [18].

Tremblay et al. [19] recently suggested that when it came to audiovisual integration, not all CI users could be grouped together. In an audiovisual fusion task, only the CI users that were unable to recognize auditory speech sounds efficiently (yet showed normal sound detection performance) were referring primarily to visual cues to process speech information. Interestingly, the results of Tremblay et al. [19] also suggest that a number of CI users, namely those who are proficient in sound recognition, can show normal-like audiovisual interactions even in situations of incongruity. This notion is strongly supported by a recent study of audiovisual segregation; namely the ability to focus on the processing of one information stream while ignoring the irrelevant and incongruent information in an audiovisual task [20]. In an auditory task with visual distractors, proficient CI users performed in a normal-like manner, while non-proficient users did not; they were in fact much more disturbed by the visual distractors that involved movement (dots, lip movements) but not, however, by color changes. To our knowledge, this remains the only study of audiovisual segregation ability in CI users.

These two investigations [19], [20] contrast with the general idea that all CI users rely more heavily on visual cues in conditions of incongruency [11], [12], [15]–[18]. More specifically, the results suggest that i) CI users can show normal-like performance in an audiovisual task, with the normal relative weight of visual and auditory cues, and ii) only the CI users that are non-proficient in highly demanding auditory tasks, such as speech identification, show abnormal, visual-oriented interactions.

Performance on audiovisual segregation tasks, however, has to be carefully assessed in order to fully confirm these conclusions. In particular, the question remains as to whether the reverse task, namely ignoring auditory distractors in a visual task, is performed differently in proficient and non-proficient CI users. The present study tackled this issue by comparing proficient CI users, non-proficient CI users and NH in a speechreading task with and without auditory distractors. In accordance with the results of Champoux et al. [20], it was hypothesized that only non-proficient users would differ from the NH. More precisely, it was hypothesized that speechreading would be affected by incongruent auditory information in NH and proficient CI, but not in non-proficient CI. The confirmation of these hypotheses would in effect also demonstrate that the results reported in Champoux et al. [20] were not due to the specificity of the task, the procedure or the stimuli used and confirm further the possibility of normal-like audiovisual interations in cochlear implant users.

Methods

Subjects

Twenty-four participants (seventeen CI users) were involved in the study. All CI users had received their implants at least one year prior to taking part in the study. The clinical profile of each participant has been described elsewhere (see [20]). All participants suffered from profound bilateral hearing loss (pure-tone detection thresholds at 80 dB HL or greater at octave frequencies ranging from 0.5 to 4 KHz) and were post-lingually deafened. The principal communication mode for all CI users was oral/lip-reading. In all participants, pure-tone detection thresholds with the CI, at octave frequencies ranging from 250 to 6000 Hz were within normal limits (30 dB HL or less). The Research Ethics Board of the Institut Raymond-Dewar approved the study and all the participants provided written informed consent. Figure 1 includes the picture of an audiologist. We confirm that this individual has seen this figure and the manuscript, and has provided written consent for publication.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Experimental procedure and visual speech recognition performance in the three audiovisual conditions.

(A) Illustration of the experimental procedure. Each visual stimulus began and ended in a static neutral position. Visual stimuli were either presented alone (baseline condition) or simultaneously with one of three types of auditory stimuli (white noise, reverse speech sounds and non-altered speech sounds). (B) Performance in the three audiovisual conditions is expressed as % decrease of performance compared to the baseline visual-only condition. In both the AV-reverse speech and AV-speech conditions, non-proficient CI users differ from both controls and proficient CI users. * : p<.05.

https://doi.org/10.1371/journal.pone.0033113.g001

Results

Visual speech recognition performance in the three incongruent audiovisual conditions is shown in Figure 1B. To determine speechreading ability with or without irrelevant auditory distractors, a 3×3 mixed ANOVA with group (control, proficient CI users, non-proficient CI users) as a between-subjects factor and audiovisual condition (AV-noise, AV-reverse speech, AV-speech) as a within-subjects factor was conducted. There were main effects of condition (F(2,36) = 114.537, p<.001) and group (F(2,18) = 9.901, p = .001). The interaction between factors was also significant (F(4,36) = 10.959, p = .001). Post-hoc Tukey HSD tests revealed significant differences between the non-proficient group and the control group in the AV-reverse speech (p = .049) and AV-speech (p = .005) conditions. There were also significant differences between the non-proficient and the proficient group in the AV-reverse speech (p = .005) and AV-speech conditions (p<.001). Post-hoc analysis did not reveal any other differences between groups (p>0.05) and as such the performance of proficient CI users was never statistically different from that of the NH controls. The performance level of every CI user was examined further in the three experimental conditions. There was a significant correlation between the decrease in speechreading performance and the proficiency to use the CI in the AV-reverse speech (r = 0.686, p = .007) and the AV-speech (r = 0.824, p<.001) conditions. There were however no significant relationships (p>.05) between visual recognition performance and the duration of deafness, the age at onset of hearing loss or the length of experience with CI.

Discussion

The present study aimed to investigate audiovisual segregation abilities in proficient and non-proficient CI users. Using a speechreading task and three types of auditory distractors, we showed that the presentation of auditory speech stimuli significantly impaired speechreading performance in proficient CI users, just like in NH participants, whereas speechreading performance was unaffected by auditory distractors in non-proficient CI users.

Traditionally, all CI users have been considered equal, and equally different from NH, in audiovisual tasks. In short, it is assumed that this population, although capable of normal integration, tends to rely more heavily on visual cues in conditions of incongruency (e.g. [11], [12], [15]–[18]. However, recent evidence from our laboratories [19], [20] highlights the importance of CI proficiency in audiovisual interaction outcomes. We suggested that whereas CI users that were proficient at speech recognition could perform at normal-like levels, those that were not would favor visual cues and show anomalous audiovisual integration. The results presented here therefore support two notions: i) that CI speech recognition proficiency is associated with audiovisual interaction outcomes in this population and ii) that several CI users, namely the proficient ones, can show normal-like performance on an audiovisual task.

Cross-modal reorganization has been repeatedly shown to occur in the profoundly deaf (e.g. [21]–[24]). In fact, in CI users, there is an activation of the early auditory cortex in the presence of visual stimuli and this activation is greater for those who show poor speech recognition abilities [25]. In addition, CI users display atypical low-hierarchical visual activity during speech recognition tasks [26]. This activity in the visual cortex is less marked and less consistent in naive than in rehabilitated CI users, suggesting that these visual cortex activations are due not only to deafness-induced plasticity, but also to brain reorganizations related to the functional learning of associations between visual cues and oral speech [25]. Therefore, different levels of auditory-to-visual reorganization in cochlear implanted deaf subjects could explain the varying audiovisual segregation abilities reported in the present study: greater cross-modal reorganization would lead to the overuse of visual information and consequently to a greater capacity to ignore irrelevant auditory cues.

Some other issues, however, could also explain the pattern of results observed across tasks and groups. First, the three audiovisual tasks arguably did not require the exact same attentional resources. Indeed, speech stimuli were more salient and more complex than noise stimuli and consequently, were more likely to capture attention. Some studies, moreover, suggest that children with CI could perform poorly on attentional tasks [27], [28], although performance might improve progressively with the use of a CI [29]. In our study, putative impairments of visual or auditory attentional processes have unfortunately not been evaluated. These capacities might need to be investigated further in those populations to better understand their implications in the present results.

In conclusion, our results strongly suggest that in terms of audiovisual interactions, proficient and non-proficient CI users should not be lumped into a single group. More specifically, we show that normal-like audiovisual interactions are possible in proficient users and we show that CI proficiency is associated with audiovisual interactions in CI users. CI proficiency must therefore be taken into account in further studies of audiovisual interactions in this population.