Introduction

While some people excel at performing two tasks simultaneously, others can only effectively perform tasks sequentially. Past research has demonstrated that the performance of multiple tasks is not hindered when both tasks are executed by different modalities, such as tasks involving vision and audition [1-3]. These studies suggested the presence of multiple processors for each modality [4]. This leads to the question of whether the source of processors for the different modalities is the same or whether the processing resource is modality specific. Some studies have suggested that each modality contains its own processing resources (i.e., that the resources are modality specific) which, to some extent, are independent from those of other modalities [5-13]. On the other hand, the observation that people cannot perform well on bimodal tasks is also supported by the evidence that processing resources are shared among modalities (i.e., modality independent), and therefore, performance of bimodal/dual tasks depends on interactions between the processing for each of the tasks [13-18].

The processing dependency in the two mechanisms involved in bimodal divided attentional tasks can be elucidated through the different predictions in our imaging study. When two tasks with separated streams of modality processing are adopted, such as a visual “what” task and a auditory “where” task [19-21], which have been suggested to have little cross-modal interaction [22], the predictions for the two mechanisms will be different. If modality specific processing exists, distinct brain areas, including stimuli-related processors, should be found for each unimodal task. On the other hand, if modality independent processing exists, then shared and common brain areas should be found for each unimodal task. Although the current research paradigm in the literature is to manipulate the task demands of one modality to examine the effect on the other [16,23-25], the manipulation of task demands may risk exceeding the limit of one modality’s processing resources, causing its interaction with the other modality. It may become a confounding variable to differentiate the two concepts, because total processing resources are limited regardless of whether they are modality specific or modality independent. As a result, the present study did not aim to manipulate the task demands of modalities.

Although past research showed activation, for the divided attention tasks, in the prefrontal areas [26-28], anterior cingulate cortex [27], and the inferior parietal lobes [28], little agreement has been reached on which brain areas are required to outperform in bimodal tasks, especially compared to those who performed well on only one of the bimodal tasks. Therefore, the present study aims to answer the question by using functional magnetic resonance imaging (fMRI). The study is important not only to determine the possibility of deciphering the neural basis of one intriguing cognitive process of human minds, but also because it shows applicability to educational training, such as instruction in how to excel at two tasks simultaneously.

Methods

Experimental setup and stimuli

The experiment contained three tasks, including visual, auditory and bimodal tasks (Figure 1). The visual task was adopted from Chiang et al. [29], consisting of either 50 green and 50 red dots or 50 yellow and 50 blue dots on a black background. Half of the dots of each color moved in opposite directions along either a horizontal or vertical axis at a speed of 4.30°/s. These dots moved at variable angles within a range of 0.14° of visual angle perpendicular to the original direction of movement. The other half of the dots of each color flashed at random locations in order to increase the task difficulty. Before testing, equiluminance was separately established for each subject by flicker photometry [30]. The task required subjects to identify which color was moving and in what direction by using a right-hand keypad to answer one of two questions that randomly appeared after the stimulus presentation. One question asked, “Which color of dots was moving direction?” (The word direction was replaced by up, down, left or right, as appropriate.) The other question asked, “In which direction were the color dots moving?” (The word color was here replaced by green, red, yellow or blue, as appropriate.) Furthermore, the questions were relevant to the stimuli. For instance, if the subjects were shown red and green dots, they were not asked about the direction of blue or yellow dots, only that of red or green dots. If they were shown dots moving horizontally, they were not asked about the color of dots moving vertically. The combination of stimuli and related questions were balanced in a random sequence such that each type of stimulus was followed by each of the possible questions in turn. Each subject’s response to the visual stimuli was categorized as either correct or incorrect based on its objectively assessed accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Schematic diagram depicting the study task.

a) The unimodal visual stimuli consisting of red and green dots moving either vertically or horizontally were presented for a specific duration. A question then appeared to which the subject responded using the keypad on the right hand and the response immediately showed on the relevant grey bar. The examples shown here are questions related to the moving direction of the red dots. b) The unimodal auditory stimuli involved the presentation of two different pitches on the left and right ear for 500 ms and icons were shown on the screen. After the presentation, a black screen appeared to which the subject responded using the keypad on the left hand in order to indicate which ear was presented with the lower frequency. The response immediately showed on the icon at the relevant location. c) The bimodal task presented the visual and auditory stimuli at the same time. After presentation of the stimuli, the subject needed to respond to the subsequent visual question with the keypad on the right hand and indicate the lower frequency with the keypad on the left hand. The sequences of visual and auditory responses were irrelevant. In the examples illustrated here, the subjects indicated that the right ear heard the lower frequency of tones and the red or green dots were moving to the left.

https://doi.org/10.1371/journal.pone.0077408.g001

The auditory task consisted of two pure tones, each randomly presented to one ear for 500 ms. One tone was fixed at the piano key f4 (349.2 Hz), while the other was randomly selected from between the piano keys g4 (392.0 Hz) and b6 (1046.5 Hz). The intensity of the two tones was adjusted to be subjectively equal in loudness for individual subjects (see procedure section below for details). The subject was then instructed to indicate which ear received a lower pitch by pressing a left-hand keypad. The response was categorized as either correct or incorrect based on its objectively assessed accuracy.

In the bimodal divided attentional task, the visual and auditory stimuli were presented at the same time to subjects. The subject was instructed to complete both tasks as accurately as possible, regardless of the order of response. The visual stimuli were constructed using the Cogent Graphics toolbox (available at www.vislab.ucl.ac.uk) and the auditory stimuli were constructed using MATLAB (MathWorks Inc.). Both sets of stimuli were executed in MATLAB on a PC computer and presented to functional magnetic resonance imaging (fMRI) compatible goggles (14.25° x 10.71° of visual angles in width and height, VisuaStimXGA, Resonance Technology Inc., Northridge, California, USA).

Results

Table 1 shows a summary of behavioral results for performance on the unimodal and bimodal tasks. The mean accuracy for unimodal tasks was significantly higher than chance level (t₁₈ = 13.54 and 13.87 for visual and auditory tasks respectively, p <0.001). Variations in duration did not affect performance because the accuracy was kept at the same level among subjects. The results were used as a baseline for the subjects who could perform single tasks well. During the bimodal tasks, each subject had two sessions of the bimodal task, which resulted in 36 sessions (degrees of freedom) for analysis. The result of each session was viewed as an independent subject because subjects’ performance in the two sessions were not identical (see Table 2). For instance, some subjects performed well in one modality in one session but performed well or badly in both modalities in the other session. Another example is that some subjects performed well in one modality in the first session and in the other modality in the next session. The results presented here were based on the performance in each session. Some subjects performed well in one modality in at least one session - eight subjects for ‘visual good’ and nine for ‘auditory good’; e.g., mean 10.1 trials for visual-good responses, SD = 2.6, accuracy = 0.73, SD = 0.07, t₇ = 13.54, p < 0.001, compared to 0.5 in accuracy) - but poorly in another; e.g., mean 4.7 trials for auditory-bad responses, SD = 2.4, accuracy = 0.52, SD = 0.07, t₇ = 0.92, p = 0.3896. In contrast, 13 participants in the bimodal task, in at least one session, performed well in both visual and auditory modalities (mean trial number = 17.4, SD = 1.6, accuracy = 0.71, SD = 0.08, t₁₄ = 13.54, p < 0.001, compared to 0.5 in accuracy).

Subjects perform two sessions in the experiment and as a result, each row has at most two “x”. Each “x” indicates what good performance was done by subjects in one session, either in one modality (visual or auditory) or in both modalities. “xx” indicates the good performance in both sessions. Some subjects have only one “x” because their performance in one session reaches the criteria of good performance (see details in the Methods) but fails in the other session.

Imaging results are shown in Table 3 for detailed statistics and Figure 2 for illustration. Compared with the control condition, the visual unimodal tasks (including both the correct and incorrect trials) resulted in activation of the visual cortices, the lateral parietal lobe, the medial frontal lobe, frontal eye field (FEF), and inferior frontal lobe (all p < 0.05 with FWE correction). Imaging results of the auditory unimodal task (including both the correct and incorrect trials), compared to the control conditions, showed a significantly activated auditory cortex including BA43 compared to the rest condition (p < 0.05 with FWE correction).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Schematic results of the statistical parametric mapping (SPM) analysis of the contrasts.

An uncorrected threshold of p = 0.001 was used for the voxel illustration. The SPM results were superimposed onto the SPM toolbox, xjView, with a template of brain structure indexed for the coordinate of the z axis according to the MNI space. Unimodal stimuli versus rests were indicated in yellow and red words for visual and auditory activation, respectively (FWE p< 0.05).Brain regions activated by visual and auditory stimuli, selected with FWE p value between 0.05 and 0.08, were in cyan and purple words, respectively, and were also depicted in yellow and red colors in the corresponding brain areas. A mask was formed to examine the bimodal task versus the average of unimodal tasks. The results shown in the green words and circles indicate three activated areas including the medial parietal lobe, BA9 and BA6, all outside the mask (FWE p value < 0.05). Finally, fifteen areas formed a ROI to examine the performance of subjects that outperformed bimodal tasks versus those who performed well in only one of the bimodal tasks. The ROI analysis shown in white circles revealed that the medial frontal lobe, FEF, BA6, BA7, and medial parietal lobe were negatively correlated with the contrast (t test, p < 0.05).

https://doi.org/10.1371/journal.pone.0077408.g002

In order to examine whether additional brain areas were recruited during performance of the bimodal task compared to the average of the unimodal tasks, a group of 10 brain areas (12 clusters) were formed as a mask (i.e., the visual and auditory tasks in Table 3). The 10 brain areas were selected from the FWE corrected p value < 0.08 in which the six mentioned brain areas mentioned above were also included. The reason we chose voxels of p < 0.08 to form the mask was to increase the statistical power (i.e., 1 - ®) and to avoid missing some voxels that truly belonged to the processing sites of visual and auditory information. Imaging results contrasting the bimodal task (including both the correct and incorrect trials) with the average of the single tasks revealed three additional brain areas outside the mask, including BA6, the medial parietal lobe and the dorsolateral prefrontal cortex (BA9), that were necessary to perform the bimodal task (all p < 0.05 with FWE correction, see Table 3 and green letters and circles in Figure 2). Inside the mask, there were no significant results of the contrast.

Furthermore, the 10 brain areas found in the unimodal tasks, along with the three additional areas defined by the contrast of bimodal versus unimodal tasks were used to run ROI analysis of the contrast between good performance in both tasks and good performance in only one task in the bimodal task. The sizes of ROIs were listed in the voxel numbers in Table 3. This helped us assess whether the behavioral differences can be accounted for by the differential activity of these ROIs. Behaviorally good performance in both modalities in the bimodal task resulted in relative deactivation of some brain areas such as BA6 (t₁₇ = -2.53, p < 0.05), FEF (t₁₇ = -1.74, p < 0.05), medial frontal area (t₁₇ = -2.02, p < 0.05), medial parietal lobe (t₁₇ = -1.76, p < 0.05), and BA7 (t₁₇ = -1.74, p < 0.05; see Table 3 and white circles of Figure 2).

Discussion

The results of our study showed that subjects who performed well on the bimodal tasks showed decreased brain activity in some visual, auditory and parietal lobes, compared to those who could only perform well on one of the two simultaneous tasks. Meanwhile, when subjects performed a bimodal task, extra brain areas such as BA6, the medial parietal lobe and the dorsolateral prefrontal cortex were required, compared with the unimodal single tasks. Our results showing the activation of distinct brain areas for unrelated visual and auditory tasks support the concept of modal specific information processing.

The visual and auditory tasks in our experiment aimed to study the distribution of processing among different modalities. Unlike the cross-model integration paradigm discussed by Spence and Driver [31], in which cues were manipulated to influence judgment in tasks, our present study does not manipulate cues before presentation of stimuli. Hence, the characteristics of the visual and auditory tasks can be viewed as independent. Alais et al. [14] found that the individual thresholds of the bimodal stimuli (pitch discrimination and visual contrast tasks) were not affected by each other, suggesting that people have the ability to perform two concurrent tasks well. This differs in part from our results, in which some subjects could outperform in the bimodal task but others could not. The discrepancy in our results and those of Alais et al. [14] may be attributed to the different criteria of task measurement. Our task adopted a criterion of approximately 70% accuracy as the index of good performance, whereas Alais et al. [14] measured the threshold (i.e., 50% accuracy) of visual and auditory modalities. Unlike a 70% accuracy requirement, an unchanged threshold in the combination of visual and auditory tasks does not mean the performance of the two tasks will always be good.

Our results showed that distinct brain areas are recruited to perform visual and auditory unimodal tasks, indicating that each modality may have its own modal specific information processing. These results are compatible with previous studies [1-3,5,6,8,10,13]. Furthermore, our results showed that additional brain areas are required for performance of bimodal tasks compared to the summation of brain areas activated in unimodal tasks. This is supported by behavioral data demonstrating that performance of bimodal tasks requires more than the simple addition of two unimodal stimuli [32-34], as well as by the physiological evidence of dorsolateral prefrontal cortices involved in bimodal tasks [26-28]. Although the current study did not directly manipulate attention, our results still reveal reliable results that match with past research about the allocation of attentional resources. For example, the medial parts of the parietal lobe have been found to be domain independent and associated with shifts in attention [26-28,35]. Furthermore, BA9 and BA6 have traditionally been considered to be related to decision-making that involves the selection of a course of action based on immediate or past information from trials [36-38]. In a recent model of “hierarchical control over decision making”[39], BA9 and BA6 were responsible for perceiving “episodic” and “sensorimotor” information, respectively, and working together to select the appropriate actions. Collectively, the evidence suggests that the additional brain areas recruited for performance of bimodal tasks may play a role in the processing of information, a role that is compatible with the concept that processing of bimodal stimuli is more efficient than that of unimodal stimuli [34,40,41].

The result that subjects who performed well in the bimodal task displayed relative deactivation in areas such as medial frontal areas, FEF, BA7, and the medial parietal lobe, compared to subjects who performed well in only one of the two simultaneous tasks, suggests that the brain needs to be deactivated to remain alert, in order to outperform two tasks simultaneously. These brain areas found in the study match the brain areas of the default network or default mode of the brain [42,43]. Although the brain only weighs 2% of total body weight, it consumes 20% of the body’s energy. As a result, the brain’s activities can be recorded even in a rest state. The primary brain areas of the default network include the dorsolateral and ventral medial frontal cortex, the posterior cingulate cortex (especially the retrosplenial cortex), the inferior parietal lobe, the lateral temporal cortex, and the hippocampal formation [44]. One of the main features of the default network is that while subjects are performing relevant goal directed tasks, the deactivated brain areas can be measured, resulting in a negative correlation between the corresponding brain areas and the tasks [45]. The default network has been considered to be related to internal cognitive activities such as episodic memory and attention shifts towards the internal state [46]. When task difficulty increases, the deactivation of the default network is linearly increased [47]. This implies that when subjects efficiently and simultaneously handle multiple tasks, they may pay attention to the information entering the brain and/or resources allocated to different modalities, rather than paying attention to external information. These findings were also supported by Tomasi et al. [48] and Esposito et al. [49] in their study suggesting that reallocation of attentional resources ensures active involvement in the ongoing tasks.

In summary, the results of the present study suggest that the perceptual processing of vision and hearing are modal specific, rather than shared or overlapping. However, the total amount of resources is limited. When bimodal tasks are executed, compared to unimodal tasks, activation of extra brain areas outside of the processing of vision and hearing are required, suggesting that interaction between the modalities is necessary to cope with the demand of bimodal tasks. The most important finding is that relatively decreased activity of some visual, auditory and prefrontal brain areas was found for those subjects that performed well in bimodal tasks compared to those who performed well in only one of the bimodal tasks. This suggests that the shift to internal states would also be necessary to handle the demand of performing concurrent bimodal tasks. Bimodal tasks cannot be like unimodal tasks in requiring as many resources as they want, because that results in a burden on the cognitive controls. Therefore, reduced resources due to reduced activity of relevant areas may be the reason for the fMRI BOLD responses; nevertheless, the performance is better in the bimodal tasks. Efficient information processing on multiple perceptual tasks and its relationship to attentional modulation needs to be assessed in greater details in further studies.

Acknowledgments

We thank the interdisciplinary MRI/MRS laboratory and J. H. Chen and C. H. Hsieh of the Instrumentation Centre, National Taiwan University for the assistance with the MRI experiments.