Introduction

It is well-known that human beings are able to associate stimuli (novel or not) perceived in their environment [1]–[5]. For example, this ability is used by children in reading acquisition when arbitrary associations between visual and auditory stimuli must be learned. Indeed, it is generally agreed upon that reading acquisition consists of two parts: on the one hand, the development of phonological and orthographic representations and, on the other hand, the establishment of associations between these two types of representation [6]–[7]. There is little research devoted to the way in which these associations come about and what there is tends to consider it as an “implicit” process triggered by the learning of letter/sound correspondences. Reading training intervention adheres to this conception [8]–[9]. However, although this type of intervention has a positive effect on reading, its acquisition generally remains slow and difficult. This means several months of formal instruction are necessary before young children grasp the logic of the alphabetic principle and use it [10]–[12].

In light of Bryant and Bradley's work (1985) [13], we assume that one of the difficulties involved in learning how to read relies partly on the establishment of associations between the orthographic representation of a word and the corresponding phonological representation, i.e., between the visual image of the word and its auditory image. In an attempt to overcome this difficulty, a “multisensory” learning method not relying only on the visual and auditory modalities as is traditionally the case, but also on the manual haptic modality, can be used. Indeed, our hands do not simply possess the motor function of moving or transforming the objects in our environment, but also have a highly efficient active perceptual function [14]–[17].

Several studies revealed the positive effects of the visuo-haptic exploration of letters in relief when learning how to read, i.e., learning how to arbitrarily associate visual and auditory stimuli. In these studies, Gentaz and his colleagues [18]–[21] evaluated the effects of two methods intended to help very young children in the understanding of the alphabetic principle. Both interventions proposed several exercises concerning the knowledge of letters (graphemes), the identification of sounds (phonemes) and the letter-sound correspondences. One letter/sound association was studied in each session. The two interventions differed according to the perceptual modalities used to explore the target letters: with the visual modality only in the classic (control) intervention and with both the visual and haptic modalities in the multisensory (experimental) intervention. The letter knowledge, phoneme identification and the decoding of pseudo-words were evaluated before and after the interventions. Among children with a standard level of letter knowledge, results revealed that the performance in the decoding of pseudo-words increased after both interventions but were significantly higher after the multisensory intervention. It should be noted that the knowledge of letters and the identification of phonemes increased similarly after both interventions.

These studies showed that the incorporation of the visuo-haptic exploration of relief letters during a training session, focused on the alphabetic principle, increased its positive effects on decoding skills in kindergarten children. Two complementary hypotheses were proposed to explain these positive effects [18]–[19]. The first hypothesis was based on the addition of the motor information associated inherently with the cutaneous and kinesthetic information generated during the visuo-haptic exploration of visual letters. This multiple coding of the letter may increase the memorization of each letter's shape [22]–[23], (for neural bases [24]–[25]) and would enable a faster activation of the multisensory representation of the letters. As a result, letter identification and then reading ability could be facilitated. The second hypothesis was based on the functional differences of the sensorial modalities involved in interventions [14], [19]. Indeed, vision is characterized by its quasi-simultaneity and is therefore more suitable for processing and representing spatial stimuli such as letters. On the other hand, listening is sequential in nature and is more suitable for processing temporal stimuli such as the sounds of speech. This functional difference could explain why young children have some difficulties in establishing the association between letters, which are processed visually, and sounds, which are processed auditorily. In contrast, the haptic modality shares characteristics with both the auditory and the visual modalities. Even though its functioning is highly sequential in nature, haptic perception is also spatial perception since the exploration in this modality is not linear and subject to a fixed order. The sequential exploration generated by the incorporation of the haptic modality would lead children to process the letters in a more analytical way than when the letters were visually presented. Taken together, the visuo-haptic exploration would help to build a link between the visual processing of the letter and the auditory processing of the corresponding sound; a “haptic bond effect”.

The aim of the present study was to examine whether the addition of the visuo-haptic exploration would lead to more efficient learning of arbitrary associations between visual and auditory novel stimuli in adults as well. This question was not trivial because the characteristics of the visuo-haptic exploration of adults are different from those of young children. First, the study of analytical processes and the integration of different object properties into a unified whole with Garner's (1974) [26] classification research paradigms have provided of some differences in the haptic modality in children and adults, mainly due to the nature of haptic exploration [27]–[29]. Because of partial and poorly organized exploratory procedures, young children proceed to inspect certain properties sequentially, and their classifications are thus based on the dimension they perceived well.

That is why, in free classification tasks, changes due to age seem to operate in the opposite direction to what is usually observed in vision, since the young preferentially make dimensional classifications and not classifications by overall similarity. But adults have reversed results: they classify more by overall similarity than by dimension in haptics, contrary to what is done in vision [30]–[32], because they give priority to the ultimate step of perceptive processing which, in haptics, is the reconstruction of the total object from its elements. Second, a visual dominance is often observed in adults: the bimodal exploration (simultaneous visual and haptic exploration) of spatial properties of shape is not more efficient (when the test is visual) than unimodal visual shape exploration. This visual dominance is not systematically observed in young children [33]–[35].

To examine whether these characteristics of the haptic exploration of adults would influence the positive effect of the addition of this modality, adults were asked to learn 15 new associations between novel visual stimuli and their corresponding sounds in two learning methods which differed according to the perceptual modalities involved in processing the stimuli. Adults used either their visual modality in the “classic” learning method or their visual and haptic modalities in the “multisensory” learning method. The performance in two intramodal (visual and auditory) recognition tests and two intermodal (visuo-auditory and auditory-visual) recognition tests were evaluated immediately after each intervention and one week after. In intramodal tests, adults were asked to find which visual (or auditory) stimulus was previously learned among five alternatives. In intermodal tests, a visual (or auditory) stimulus was presented to the participant who was asked to find its matching sound (or visual) stimulus among 5 alternatives. If the addition of the visuo-haptic exploration is efficient in adults as well and thus helps them to learn more effectively the arbitrary association between visual and auditory novel stimuli, performance in the intermodal recognition tests would reveal traduce this positive effect. The ability to recognize the associations would be above the chance level after both interventions but higher after the multisensory method than after the classic one. If this positive effect is indirectly due to better memorization of visual stimuli (as we hypothesized in previous studies), participants would show a better ability to recognize the visual stimuli after the multisensory method than after the classic one. In all cases, participants would show a similar above-chance ability to recognize the auditory stimuli after both methods. Furthermore, we expect that with a delay between the learning phase and recognition tests, performance will remain stable after the multisensory training method and decrease for the classic one, according to time-dependant consolidation found in some motor memory tasks [36] and the standard forgetting curve in the visual memory.

Results

Discussion

This study examined whether the addition of the haptic modality would lead adults to more efficient learning of arbitrary association between visual and auditory novel stimuli. In addition, we hypothesized that this enhancement could be due to a better memorization of shapes. To test these hypotheses, adults were asked to learn 15 associations between novel visual stimuli and their corresponding sounds with two learning methods which differed according to the perceptual modalities involved to process the visual stimuli. The participants used their visual modality in the “classic” method and their visual and haptic modalities in the “multisensory” method.

The first result was that the performance in the visuo-auditory recognition test was above chance after both methods but was better after the multisensory learning method than after the classic one. The addition of haptic exploration of visual novel stimuli seems to help adults to associate more shapes and sounds than does visual exploration only. This result was consistent with the results observed in ordinary children, as reported in the introduction. But, contrary to our hypotheses, the performance in the audio-visual recognition test (i.e., the reverse direction) was similar after using both learning methods. Furthermore, an asymmetry appeared in the intermodal recognition tests for the multisensory group. Indeed, participants recognized more associations in the visuo-auditory recognition test than in the audio-visual one. It should be noted that asymmetries were often observed in crossmodal tasks like between vision and touch in infants, children and adults ([37]–[39]) and their explanations are still in debate (for a review [15]).

According to Ernst and Bulthoff [40], one determinant of where crossmodal convergence of object information occurs is which modality provides the most accurate information about objects. Usually, humans obtain most of their information about objects from vision for which shape is the most salient attribute. In our research, because we used arbitrarily related multimodal information (shapes and sounds), both visual and auditory modalities involved in learning delivered different relevant information (spatial and temporal). Thus, the crossmodal memory could be maximized (“sensory combination”) and participants were able to recognize clearly shapes from sounds and vice versa. On the other hand, when we added the haptic modality in the training phase, it is possible that redundant signals about spatial information on the visual stimuli, provided by both visual and haptic modalities, increased the percept reliability (“sensory integration”). First, it could explain why the multisensory group obtained better performance than the classic one. Secondly, because of competition from more salient modality-specific information (spatial vs. temporal), it could generate an asymmetric crossmodal performance.

The second set of important results showed an equivalent mean number of visual stimuli recognized by participants after both training methods. The haptic effect observed in the intermodal task cannot be simply explained by a better memorization of the visual stimuli. However, Pascual-Leone and Hamilton assume that relevant inputs from senses are exploited to execute particular processing tasks successfully [41]. It could be speculated, because of the nature of the task (to recognize learned shape among unlearned ones), that the visual modality alone is recruited and provides enough reliable spatial information to perform the task as accurately in both learning groups. We could also hypothesize that the shape percept is improved by additional information collected via the haptic exploration, which may improve recognition speed. Then, the intramodal recognition test used in the present experiment constituted too global a measure of the shape knowledge because it did not take into account the speed of shape recognition.

Finally, we noticed a slight improvement of performance in the visual intramodal recognition test after a week of delay. This result was not expected because we believed that performance remained stable in the multisensory training and decreased in the classic one because of the potential effect of the haptic exploration on the memorization of shapes. This result is probably a false positive because, on the delayed recognition test, the participants of both groups benefited further from having seen the trained shape a second time during the immediate intermodal recognition test.

In conclusion, the present study underlined the positive effects of the addition of visuo-haptic exploration in learning of arbitrary associations between novel visual and auditory stimuli in adults (for review about haptics in education [42]). Although the mechanisms of its action are still in debate, haptic exploration seems to play a role in “bonding” between visual and auditory stimuli in young children as well as in adults, whether or not the characteristics of visuo-haptic exploration of adults are different from those of young children.

Methods

1. Participants

Thirty monolingual French adult students took part in this experiment (Table 1). The participants in the two learning methods were matched on each of the following criteria: age and Raven test (t test: p>.25). There were 15 adults in each group. The present study was conducted in accordance with the Declaration of Helsinki. It was conducted with the understanding and the written consent of each participant which was obtained and was approved by the local ethic committee of the LPNC (CNRS and University of Grenoble 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of participants in each training group.

https://doi.org/10.1371/journal.pone.0004844.t001

Supporting Information

Acknowledgments

We thank the adults who took part in the experiment, and Bernard Ans, Sébastien Caudron, Clément Fredembach, David Hope, Marie Lenormand, Sébastien Pacton, Pierre Perruchet and Carole Peyrin for comments on the manuscript.